Spifftastic,

 

Ascension 2 Development

I released the Ascension 2 Live Wallpaper on Google Play last month, so it’s probably high time I wrote about it. I’ll just run through a few topics and see where that goes.

For those not sure what Ascension is, you can first click the link to the store page, above, and that should give you a good idea of what it looks like. If you don’t know what a live wallpaper does: it displays a wallpaper on the Android home screen (the launcher), and typically the wallpaper animates or provides information or some other feature. It might render a model, maybe a tree, maybe an aquarium. Ascension displays bars that change color over time and react to touch. I mean for Ascension to stay subtle, more so than most live wallpapers, but it still allows users to customize it enough to make it their own.

I originally built Ascension for myself, but I’ll try to explain why I built Ascension 2. I still made it for myself, but its purpose changed a little this time around.

Motivation

Ascension’s actually pretty old now. I released it back in 2010 — when live wallpapers first appeared and various spam-developers released tons of them. Some featured sphere-mapped Android robot models floating in front of a background, every national flag, and other garbage.1 I still used an Android phone at the time and wanted a live wallpaper, so I made the first version of Ascension to appease my desire for a good live wallpaper.

After a while and some updates, I decided I no longer wanted to develop for Android. Android grew fragmented between various OS versions and hardware specs, and to an extent it remains that way today. Not so much OEM skins anymore, but they’re there. I ditched my Android phone for an iPhone and left Android behind while it went through its growing pains with tablets (recall the Motorola Xoom and how it failed to deliver on its features). Google later recognized that performance on Android sucked, so they tried to fix most of that with Android 4.0. Anyway, Android sucked at the time.

Come 2013, the Android platform had settled, tablets became usable (though they still lack good apps), and the fragmentation issue slowly began to sort itself out when Google stopped making OS upgrades necessary to benefit from app improvements when they shoved a lot of Android into Play and moved their apps to the store.2

Background aside, Android sucked less, I had a Nexus 7, and I figured I should try to re-learn Android development and forget most of what I learned. Ascension sat in its little corner of the Play Store, horribly out of date and ugly and generally unpleasant to use.3 That in mind, I figured I should bring Ascension back up to speed with the current Android. So, Sometime early in December 2012, I started to write Ascension 2. I initially wrote this reboot in Java. Most people will pause and think, “Well, obviously, you use Java to write Android apps.” I think “But wait, there’s more!” sounds appropriate here.

I wrote the first version of Ascension 2 in Java. I never released this version, neither to testers nor the Play store. It worked very well, but I stopped writing it when finals started up. I still had my degree to finish at the time, so I put it on hold. When I returned to the code afterward, I decided I didn’t want to use Java, so I killed that version of Ascension 2, went back to my iPad, and ignored my Nexus 7 except as a curiosity until September of this year. Its battery ran down to nothing several times during that period.

In early September, I still hadn’t found a job, and needed something to keep me busy. I couldn’t let myself just tinker with code and various projects, so I looked at the broken build of Ascension 2 I had on my Nexus 7 and decided I’d finish it. Except not in Java. I’d made my decision there, and I would find another language that didn’t piss me off. I only considered two options for alternative languages: Mirah and Scala.

Mirah looked like Ruby, which as far as I know is intentional. It’s not like JRuby, where Ruby runs on the JVM. Instead, Mirah borrows some of the syntax and compiles to JVM bytecode. I needed to find a language that compiled to JVM bytecode, otherwise it’d be difficult to compile to dex, so Mirah met that requirement. The problem with Mirah is that it’s still a bit iffy right now, and although I want it to become a popular JVM language, it doesn’t feel stable. So, I dropped it as a possible choice soon after I looked at it.

Enter Scala, which for a long time bore the title of “the language that confused me.” In retrospect, I don’t know why I found it confusing, but I’ll assume I just didn’t give it a good look. Scala is also like Ruby in that it tends to result in expressive code without the verbosity of Java, though the similarity ends there. It’s also both imperative and functional, and though nobody should listen to me when it comes to functional programming, I think having it leads to generally better code, assuming you avoid side-effects and otherwise write deterministic functions.

After I toyed with Scala for a while, I decided it had everything I wanted and, at the very least, I could use it as a less-verbose Java.4 In the best case, it would help ensure I wasn’t doing anything horrible. I still do horrible things, of course, but Scala let me get away with writing a lot of code that would cause me pain in Java.5

Scala has its ups and downs, particularly when it comes to Android development. Scala’s major upsides, those that let you write better code, are that it avoids verbosity, it provides pattern matching, it allows you to write both functional and imperative code, it supports anonymous functions, traits that allow mixin-like composition, and plenty of other features. This all leads to code that tends to be more expressive and less error-prone. As a short example, it doesn’t take too much work to get to a position where you can write code like this:

import android.app.Fragment
import android.view.{View, ViewGroup, LayoutInflater}
import android.os.Bundle
import android.widget.Toast
import scala.language.implicitConversions
import net.spifftastic.view.util.implicits._

class MainMenuFragment extends Fragment {
  override def onCreateView(inflater: LayoutInflater, root: ViewGroup, state: Bundle): View =
    inflater.inflate(R.layout.main, root, false)

  override def onViewCreated(root: View, state: Bundle): Unit = {
    super.onViewCreated(root, state)

    (root withView R.id.new_game) {
      _ onClick Toast.makeText(getActivity, "Poppy", Toast.LENGTH_LONG).show()
    }
  }
}

One small downside: I encountered trouble when I tried to use Scala’s standard library with Android. In particular, if you include the entire standard library, you will easily exceed the Dalvik VM’s 16-bit method limit per dexfile.6 ProGuard solves this but, as usual, it takes some work to configure ProGuard. Rather than wrestle with ProGuard yourself, though, I’d recommend anyone interested use pfn’s Android SDK Plugin for SBT. It’s hard to adjust to SBT, but it’s great when things start to work, and pfn’s plugin will handle most of the heavy lifting for projects. If you decide to go with Scala, though, you’ll more or less need to use Scala anyway, so just get comfortable with it.

After that, I set some goals for Ascension 2. All of the goals were set with the idea to take something that exists and learn to use new APIs to implement it. The goals:

  • Aim for almost complete feature parity with Ascension 1.

    This sounds obvious, but the “almost” there is important. There are features in Ascension 2 that still haven’t made their way into the release build. I’ve delayed the first update for one particular feature, as well. I really want what I’ve planned to make it in early on, but also wanted to get it into users’ hands sooner. As a result, Ascension 2 does not have one setting that Ascension 1 had: you cannot specify a custom bar color. That feature’s coming back in a different form, but more on that when I finish the update. It’s not a small addition.

  • Design the app for tablets first.

    Ascension 2 has no phone-specific UIs, so you see the the tablet layout on all devices. There are minor differences that depend on the screen’s width in DIPs, but otherwise Ascension uses the same UI everywhere. All devices and tablets get both a settings pane and a preview pane to view their changes. Only a few settings require you to open a dialog to select something, and otherwise the settings app shows every setting as you change it.

  • Allow users to save their configurations.

    Users of Ascension 1 requested this often enough, but I designed Ascension 1 in such a way that it would’ve been difficult to implement (meaning Ascension 1 had bad code). Ascension 2 has this, and although I doubt it sees much use, I’m sure someone is grateful that they can save a configuration and load it again later.

  • Implement the renderer with OpenGL ES 2 as a minimum.

    GL ES 2 just lets me move more work off to the GPU and in doing so I’m made less dependent on hacks to implement certain features. For example, I implemented brightness in the shader, whereas previously I had to account for it when the renderer generated bar colors. The code gets smaller and easier to maintain as a result.

  • Target Android 4.x and up.

    I decided to only target Android 4.x because I just don’t want to support older devices. It’s not fun, it leaves me hamstringed if I want to adopt new APIs since I then have to try to maintain compatibility with older APIs, and it overall just limits what I can do. If I had continued to target Android 2.x, I would no longer have access to PreferenceFragment, for example, a class crucial to Ascension 2’s design. Rather than limit myself and target older devices, I decided to target what let me make the app I wanted.

None of this should surprise anyone, or at least not developers. Put short, I wanted a way to get back into Android development, I wanted to use a language that didn’t suck, and I wanted to build an app that felt like it fit in on Android 4.x. As such, it seemed right to take an app that I made for myself and build it again, improve it, and do it right this time. The first problem I had came down to how to design something for a tablet.

Tablet UI Layouts

As I mentioned above, when I designed Ascension 2’s user interface, I made it for tablets first. Also, I have no Android phones.7 When I test on a phone, I either borrow family members’ phones or ask my testers to run something on their phones. So it makes sense that I design the UI for what I’ve got on hand.

First off, because Ascension is a live wallpaper, it has only one main user interface, aside from the wallpaper itself, to worry about: the settings activity. Ascension 1.x’s settings had no preview and required a lot of taps to change things, and you couldn’t preview your changes. Users needed to poke around in the dark and see how things looked for each change. I decided early on, then, that I needed to fix this.

Ascension 2 displayed on a Nexus 7 in landscape orientation.

Ascension 2 — Landscape Orientation, Nexus 7

Above is a picture of Ascension on a Nexus 7, in landscape. This is the UI you get on all devices. The only difference is portrait orientation, which sees the preview pane placed at the top of the screen.8 The preview pane, in landscape, is always on the left. This is to put it out of the way of your right hand, which — since most people are right handed — means you can scroll, swipe, and otherwise interact with the settings without using your left hand, or even having to do anything other than move your thumb.9

The largest change to the settings UI simplified how users interacted with the settings pane. Early on in development, the activity started off with a list of the settings pages and the choice to save or load a config. In short, you had a list:

  • Bars10
  • Colors
  • Save Config
  • Load Config

You could tap any of these to get to the actual settings or work with configs. This sucked. It meant that, in order to get to another settings page, you had to tap the back button then tap the entry for the other page. This persisted for a while, since I had to get the settings hooked up before I could find a better way to display them. Once I’d had a few things hooked up to see how I wanted them to work (I’ll go into that in a moment), I pulled out the basic list and replaced it with a ViewPager and ActionBar tabs.

This was much simpler than I’d expected and made the settings app easier to navigate. The downside is that there are two ways to get between settings pages, but they’re both easy to use: swipe the page or tap on a tab. No need to press the back button. The only problem I had is that the normal FragmentPagerAdapter generates names for the fragments it provides. This makes it difficult to communicate with specific fragments, so I reimplemented it in Scala. In reality, this isn’t a huge deal and implementing a PagerAdapter shouldn’t take too long, as with most basic adapters.

The preview pane itself doesn’t need too much explanation: it allocates its own renderer and displays everything the same as the live wallpaper normally does. The only difference is that you cannot change its offset by swiping, since it doesn’t have multiple pages of content.

The important part of the new settings panes, however, involved showing changes to settings as they happened. Rather than go the easy route and embed preferences’ layouts in a dialog (using DialogPreference — also known as the lazy coder’s preference), you just place them in the preferences’ layouts. It’s not hard to implement this and it makes it a lot more fun to customize Ascension’s settings. There’s no tapping back and forth between views to see what’s changed, you just change it, and you can see from the preview pane what’s changed.11 For me, that proved that the preview pane worked.

Configurations

As I mentioned above, one of the main goals for Ascension 2 was to let users save their configurations. Each saved configuration is written to a JSON file. They look like this:

{
  "use_touch_color": false,
  "bar_ping_lifespan": 15,
  "shimmer_speed": 0.20000000298023224,
  "use_uniform_height": true,
  "use_bar_pings": true,
  … and so on …
  "bar_count": 100,
  "flip_bar_mode": "even"
}

I had no plans to necessarily make them human-readable, and I wouldn’t say the JSON makes them accessible in that sense, but it does allow you to tweak the files in a text editor if you really wanted to. This also allows you to share configurations, though I don’t expect any users to do this. At any rate, it’s easy to save a config. The challenges all involve how Ascension 2 loads configurations, displays their previews, and deletes configurations.

It’s simple to load a config, apply its properties, and persist the values, but it’s complicated to refresh all the preferences’ views once done. You have to set every preference’s current value to the newly persisted value. Preferences do not automatically refresh their views, which is sensible, there’s no reason for most to ever check their values except when initialized or changes are persisted. That said, sensible or not, you’re required to do it for them, similar to how one must stimulate an abandoned kitten’s bowel movements.

As such, I have to notify all PreferenceFragments in the settings activity to refresh their values. In Scala, this just meant I had to write a new trait and include it with each PreferenceFragment subclass:12

trait RefreshablePreferenceFragment extends PreferenceFragment {
  import RefreshablePreferenceFragment.TAG

  def refreshPreferences(): Unit = {
    implicit val sharedPreferences = getPreferenceManager.getSharedPreferences

    Setting.values foreach { key =>
      val keyString = key.toString

      findPreference(keyString) match {
        case tsp: TwoStatePreference =>
          tsp.setChecked(sharedPreferences.getBoolean(keyString, tsp.isChecked))
        case sbp: SeekPreference =>
          sbp.setProgress(sharedPreferences.getInt(keyString, sbp.getProgress))
        case lp: ListPreference =>
          lp.setValue(sharedPreferences.getString(keyString, lp.getValue))
        case csp: ColorSelectorPreference =>
          csp.setColor(ColorPreference.getColor(keyString, csp.getColor))
        case _ =>
          Log.d(TAG, s"Unhandled preference change: $key")
      }
    }
  }
}

With this, it’s possible to then call refreshPreferences on each PreferenceFragment that supports it. In the case of Ascension, this means all of them. This excludes preference types not used by Ascension, but it takes little effort to add more as needed. Only the ColorSelectorPreference type requires special handling due to how I’ve encoded colors preferences’ values, but it’s still not as difficult as it would be otherwise.

Displaying a configuration preview required building a little offscreen renderer that simply took a GLSurfaceView#Renderer, let it draw a frame, and then returned a Bitmap from the renderer. That’s all easy to do, but doing it asynchronously isn’t as simple, seeing as the settings app generates configuration previews and caches them asynchronously.13 Ordinarily, nothing happens to cause an issue — things only fall apart when the activity dies while there are tasks running to generate the previews, or when the configuration changes. In either case, though, the activity is destroyed, so the fragment is as well.

When the config picker fragment dies, it tries to pull down the offscreen renderer and destroys its context. Normally, this should be pretty easy — synchronize on the renderer, wait for it to be free, and then destroy it. The problem here is that you end up with a deadlock, mainly due to how I’d originally designed the renderer and config preview generation code. One bit would grab a resource and wait on the renderer, while the fragment would grab the renderer and wait on the resource. This meant that if previews were still being generated when the fragment went down, the entire app froze.

In the end, I solved this just by never locking the renderer and instead atomically setting a flag. If the renderer isn’t in use, it’s locked and brought down immediately, which is what usually happens. If it’s in use — still rendering — then the flag is set and when the next render is complete, the renderer is torn down and all further use of it raises an exception (which is caught). By doing that, the renderer can continue to do its thing just long enough to get torn down without the need to throw a wrench in the cogs.

Aside from that, rendering the config previews is fairly straightforward. The only thing that might sound odd is they’re 16-bit BGR bitmaps, but that’s to conserve memory. The previews are small enough that it makes sense to limit their depth.

The last issue with configs gave me a headache, because it involved Android’s media scanner. A quick overview for anyone who hasn’t suffered it: when you plug in an Android device and grab files off of it, the files you see are those on the external storage that Android already scanned. If an application writes a file to external storage, the user may not immediately see the file in external storage — possibly not for a long time, depending on when the scanner runs next. You can usually force it by rebooting, but you don’t really want to ask a user to reboot to pull a file off the storage. So, to make it visible right away, the app tells the media scanner to go ahead and scan the file.

When Ascension creates a config file, it does this. When Ascension deletes a config file, it also does this, and then it tells Android to remove the file once it’s scanned. I do it this way because, if the app deletes a file, it’s still visible in the storage afterward because Android hasn’t scanned and noticed that it’s gone. To tell Android to remove a file, you need the file’s URI in the media content provider. To get the URI, however, you have to scan the file. So when Ascension deletes a file, it performs these steps:

  1. Scan the file and listen for when the scan completes.
  2. Get the URI once notified that the scan has completed.
  3. Delete the file.
  4. Tell the content provider to remove the URI from its database.

This sounds simple, because it is, but it took me a surprisingly long time to do things this way. I figured there has to be a better way, and I assume there still is, but I haven’t found one. As such, the current method to delete one or more config files uses this odd route through the media scanner. It works, I just wish I didn’t need to go through the media scanner for it to work. Still, it keeps things clean on the storage side.

Rendition

At least one person wanted me to discuss rendition in Ascension, but most of it bores me to death because there’s nothing particularly unique about it. Everything is drawn using OpenGL ES 2, but for the most part, the data used by OpenGL rarely changes. The only exception is the colors, but otherwise nothing unique.

Ascension’s renderer uses two fixed-size buffer objects: a vertex buffer and an index buffer. Both allocate the maximum size possible for their buffers. The index buffer is initialized once and never touched again,14 since there’s no need to ever modify it. The indices for bars never change, only the number of bars and their vertices. The vertex components are stored in different segments of the vertex buffer (i.e., not interleaved), so that way only components that need updating will be updated — in normal usage, this means the renderer only updates the color component.

Continuous rendition to the wallpaper engine’s surface is handled by a GLSurfaceView instance because it works well and allows you to queue up events on the thread that handles the EGL context, renderer, and so on. I tried to avoid this, originally, in favor of used ZeroMQ via JeroMQ for sending messages to the renderer, but the problem there is that I’m not directly in control of the GL thread the view manages, so I end up with issues where one socket lives on, which makes me unable to kill the ZeroMQ context. This causes the service to lock up, and it’s just kind of a mess.

Aside from that, Google forbids the use of sockets on the main thread in Android 3.x and up, so you now have to deal with two problems: how you synchronize socket access and how to kill both the context and sockets with AsyncTasks, but in order. You can use a serial executor, but the problem is that one socket is being closed from the GL thread. To ensure order, you have to push the work off to the executor. Meanwhile the service might have already tried to kill the context, causing it to block the executor’s thread. So, the socket is never closed while the context waits for you to close it. Ultimately, I ditched ZeroMQ and used GLSurfaceView’s queueEvent method:

override def onTouchEvent(event: MotionEvent): Unit =
  /* Copy and dispatch before the event gets recycled. */
  if (event != null) {
    /* For some reason, I've received null events, so check for null events. */
    val eventRunnable: TouchEvent = TouchEvent.Cache.allocate()
    eventRunnable.renderer = _renderer
    eventRunnable.event = MotionEvent.obtainNoHistory(event)
    _surfaceView queueEvent eventRunnable
    super.onTouchEvent(event)
  }

Another point to note here: events are cached (see: TouchEvent.Cache.allocate(). This avoids triggering the GC for most situations, since the cache eventually saturates (usually at around 12 to 16 objects — the cache in use above has no upper limit, though one can be set) and no more TouchEvent objects are allocated. This might be overkill when it comes to avoiding the garbage collector, especially given that the GC on Android has improved quite a bit (it’s difficult now to get it to cause frame skips). Still, if possible, I prefer to avoid the GC at all costs.

The above example applies as well to all other events: Ascension configuration changes15, offset changes, touch events, pause and resume events, and so on. Most events are handled as they’re received by the renderer, the only exception is the config event. When a config event is received, it includes the preference key that changes. Ascension takes the key and keeps a set of all changed keys to avoid pulling data out of preferences before it needs to. As such, all config changes are coalesced into a single config delta that is applied at the start of the next frame.16

There’s not that much else to say about rendition. Inside the renderer is a single fixed-step logic loop to keep the bar state updated and a check to delay the frame if not enough time has elapsed since the last frame.17 When the frame is actually drawn, the renderer loads any resources it needs to. Then, the bar state is passed to a color generator which the renderer uses to write the colors for visible bars (plus a few to account for bar overlap) to the vertex buffer. Then, only the visible bars are drawn. Repeat until the engine or service dies.

An Abrupt End

Since I continue to work on Ascension 2, I’ll have more problems to write about. I’ve probably forgotten a few problems that gave me trouble just because I had to block them out of my memory. The difficulty I had when I started to develop Ascension 2 mostly stemmed from using new tools, a new language, and unfamiliar APIs (for example, prior to this, I had never touched fragments). Still, I’ve had a lot of fun so far, and am happy with how far Android has come. It lacks good apps, but I feel well-equipped enough that I can make the apps I want.

Anyhow, Ascension 2 is on the Play Store now. That’s it for now — I’ve droned on for nearly 5,000 words about it and avoided even one mention of centipedes. I’ve got an update to finish and another app to write, so back to work.18


  1. This situation hasn’t improved much. Ascension 2’s competition today includes “twerk” wallpapers, boob-jiggle wallpapers, the many tons of seasonal- and holiday-themed wallpapers that get crapped out on a regular basis, and so on. I believe there is also at least one almost-furry-porn live wallpaper out there as well. Make of that what you will. [return]
  2. The latter happened fairly early on, though. I remember Gmail getting updates via what was then the Android Market (which I still think is the better name — seriously, who the hell came up with “Google Play”?). [return]
  3. To change the bar count and see what it looked like, you had to tap a preference to open a dialog to tweak a number or SeekBar, then close the settings, see if it was what you wanted, and go back to tweaking settings. Repeat this for most settings and you had a ton of things you could customize — Ascension’s main draw — and a lot of taps to go back and forth between settings and a preview. [return]
  4. At least one Scala user will cringe at that. [return]
  5. For example: tail recursion works in Scala. In fact, you can slap an @tailrec annotation on a function and the compiler will emit an error if you fail to write a tail-recursive function. If that doens’t make you immediately happy, you’re a horrible person. [return]
  6. Android Issue 7147 [return]
  7. The apps I use and enjoy all live on the iPhone and iPad, so it makes little sense for me to use an Android phone. Much as they’ve improved the OS and devices, I still enjoy iOS more. There’s also the problem that all Android phones try to top the human head in size. Where apps are concerned, I figure I’ll get what I want on Android as soon as I make them myself, so until then, I’ve stuck with my iPhone and iPad for day to day use, and my Nexus 7 — the only Android device I own — sticks around as that thing I grab when I want to see if Android has any “killer apps” yet. It doesn’t. [return]
  8. Since I usually have my thumbs near the bottom of the screen on a tablet, I decided the part that sees more interaction would go on the bottom. In earlier layouts, I’d placed the preview pane at the bottom of the screen to keep the tabs and the settings connected, but this annoyed me after long enough. [return]
  9. I may add a setting specifically to enable a left-handed layout later, though it would only swap the preview and settings panes. [return]
  10. Now the “Appearance” tab. [return]
  11. The exception to this is the multiply blend mode, which unfortunately is both difficult to explain and might confuse people who don’t already know how multiplying two colors works. So, someone might enable multiply blend mode, see everything go black, and just assume it’s broken. In reality, multiply only works with light colors precisely because it multiplies the background and bar colors together. So, it also assumes you know basic arithmetic. [return]
  12. Had I used Java, this would likely be a subclass of PreferenceFragment that sat between PreferenceFragment and each of its subclasses in Ascension. The code itself probably wouldn’t be too complicated, but it would be less pleasant than writing with RefreshablePreferenceFragment. [return]
  13. Using an LruCache, the config picker fragment only stores so many configs in memory before it dumps them (8MB of bitmaps, specifically). They’re never written to storage, so the previews are just re-drawn as needed. I’d change this behavior, but it’s not likely to be an issue, and I’d rather not pollute even temporary storage with a ton of bitmaps. [return]
  14. Except when the app loses the EGL context, either because the configuration or underlying surface changed. [return]
  15. An Ascension configuration change is different from a regular configuration change in that it only affects the current Ascension settings. It’s not the sort that causes an activity to restart. [return]
  16. The config delta is just a bitset, since there’s little need to store actual strings or symbols or objects. This keeps the delta relatively inexpensive and avoids unnecessary allocations on the GL thread. Beforehand, it was just a regular set of Scala Enumeration values, which was fast but resulted in allocations to put objects into the set. A bitset stood out as a good alternative to avoid allocations, since each Enumeration value gets an integer ID, which can be poked into the bitset. [return]
  17. It’s important to not simply return early when doing this, however, since GLSurfaceView will swap buffers even if your renderer does nothing. Instead, sleep for a short amount of time. This is a bad way to do things because it blocks the view’s GL thread, but you will usually only have one active GL view at a time, meaning only one renderer will block the thread. If you have multiple views, you should consider a different way to handle frame limiting. [return]
  18. If anyone wants me to go into more detail about something, shoot me an email and I’ll either reply directly or shove another post up on here. My address is over on the Contact page (see: menu bar). [return]