OTA: Episode V : XZ strikes back

Integrates xz into pbzx, and supports those uncompressed blocks you find here and there

Jonathan Levin, (@Morpheus______), http://newosxbook.com/ - 07/28/2016

About

The new HomePod (AudioAccessory1,1) OTA which Apple released (for whatever reason) is slightly incompatible with my old pbzx implementation due to quite a few uncompressed (16M) blocks they have in the middle of the XZ stream. I normally handled this with a shell script, but figured I can just integrate xz-devel's lzma decompressor directly. So here it is. You won't need to run xz manually anymore (though you will need liblzma, which you can easily get). This has been tested on all OTA updates (iOS/WatchOS/TvOS/PodOS/MacOS), but if something doesn't work for you, TELL ME.

Because lzma's not my source, I added my mods there with an #ifndef NOJ.

This works on MacOS and Linux. Sources will compile if you have liblzma (https://tukaani.org/xz/, or the Linux xz-devel package).

Example

root@Qilin (/NewOSXBook/.../payloadv2)  cat payload |./pbzx > OTA               12:46
Flags: 0x1000000
Chunk #1 (flags: 1000000, length: 3343604 bytes)
OK!  (16777216 bytes)

Chunk #246 (flags: 1000000, length: 3748084 bytes)
OK!  (16777216 bytes)
Chunk #247 (flags: 461405, length: 1190692 bytes) 
OK!  (4587520 bytes)
... 
# Warnings are totally fine - these are the uncompressed chunks
Chunk #95 (flags: 1000000, length: 16777216 bytes)
Warning: Can't find XZ header. Instead have 0xc83bbeb5(?).. This is likely not XZ data.
Chunk #96 (flags: 1000000, length: 16777216 bytes)
..
Chunk #246 (flags: 1000000, length: 3748084 bytes)
OK!  (16777216 bytes)
Chunk #247 (flags: 461405, length: 1190692 bytes)
OK!  (4587520 bytes)

#
# Use ota (from previous articles) to unpack, or search
#
root@Qilin (/NewOSXBook/.../payloadv2) # ota -l OTA                     12:48
.file
.mb
Applications
Applications/AirMusic.app
Applications/AirMusic.app/AirMusic
..
..
usr/standalone/firmware/oscar/coreos-oscar2.image
usr/standalone/firmware/oscar/memtest-oscar2.image
usr/standalone/firmware/peppy/B238_PeppyMA_App.bin
usr/standalone/firmware/peppy/B238_PeppyMA_Merge.bin
# You can check integrity of the OTA by -l and ignoring output:
# If there are any issues, they'll be reported to stderr (2>)
root@Qilin (/NewOSXBook/.../payloadv2) # ota -l OTA > /dev/null                  12:48

Other articles in series

I
II
III
Episode IV
Episode VI
Episode VII
Episode VIII (the conclusion)

Binary pack

A tar with MacOS (x86_64) and Linux (ELF64) binaries and sources

Sources

Updated pbzx.c Modified 02_decompress.c from lzma to decompress byte stream instead of file

License

  • All yours. A nice greet would be nice if you end up github'ing it like some people do.

    • Greets

  • AAPL: For just giving up on encryption and providing non-differential OTAs for devices whose filesystem would be almost impossible to obtain otherwise
  • https://tukaani.org/xz/ - for making XZ freeware

    • P.S.

  • Yes, I cover the (very basic) file format in the upcoming MOXiI 2nd Ed (Volume II, though)
  • Yes, Volume I is on track for its scheduled release , as per The official announcement.