The Twitter tirade started after we saw yet another “Apple Blue Line Bar Graph Better Than Android Gray Line Benchmark”. The A12 is more powerful than any Android, and the A13 will beat that!
But here’s the problem.
I truly believe Apple chips are silly powerful, but for the last four years, Apple really hasn’t let us touch that power. I shared my rendering experiences again, comparing the iPhone XS against the iPhone SE. In iMove, the iPhone SE continues to render video faster than the XS.
[…]Rendering the same video, the OnePlus is a LOT faster at the task than the more expensive XS. The OnePlus also delivers a final video at twice the bitrate of the iPhone (which does look better to my eye). Better quality, twice the size, in two thirds the time.
The common wisdom is that Apple’s A series chips are considerably faster than their Snapdragon counterparts, and I, too, have highlighted that wisdom here on OSAlert a number of times.
However, if we leave the world of synthetic benchmarks and Apple’s terrible bar graphs behind and start looking at real-world performance, the common wisdom doesn’t seem to hold up. When even an outdated iPhone SE beats another iPhone that’s years newer and four times as expensive, you know something’s up.
Performance is more complicated than a synthetic benchmark that can be gamed or Apple’s entirely meaningless bar graphs.
I am just a happy MotoG5 user. It is enough for my needs.
Last time I checked, the purpose of video encoding was to produce _smaller_ files, not larger ones. ‘Twice the size, in two thirds the time’ sounds pretty pale in that regard.
This article talks about somewhat perceived video quality. It also compares completely different encoders that are claimed to produce somewhat the same quality: ‘“Standard quality” on Luma is pretty close to “High Quality” on PowerDirector’, like, really?
Apple’s bar charts whatever, couldn’t care less, but this article doesn’t really add any real insight into video encoding performance between these devices.
> Performance is more complicated than a synthetic benchmark that can be gamed or Apple’s entirely meaningless bar graphs.
Yes, Thom, and it is also more complicated than some random asshat encoding a video with random encoders and settings. You would need a controlled experiment, e.g. with target file sizes, and quantitative measures of video quality.
I guess his point is, what is the end user experience like for a use of pro video tools on the different devices. It appears that every combination he tries ends up with the IOS experience being slower. I mean he’s using apple hardware, apple created operating system and apple applications and its still slower than the rough equivalent on Android. So its a bit misleading to talk about CPU or GPU performance, sure, but its totally fair to say that a user of an ios device doing pro vido tasks doesn’t see a large performance gain over an Android device doing similar things. The Bar chart of raw cpu/gpu performance doesn’t capture the end user experience.
Bill Shooter of Bul,
It seems like the author is saying two different things, the gains shown by synthetic benchmarks haven’t translated to improvements over android in his testing, and IOS restrictions are impeding sensible work flows for power users. At this point though, I think most people going into the apple ecosystem should realize they are getting a less flexible product from the get-go. This may not be a problem for typical users with basic needs, but IOS constraints become increasingly problematic when apple tries to classify their devices as professional tools.
Yeah this is a fair point. Obviously the CPUs and GPUs are extremely fast and much faster than the SE and the Androids. Seems weird to doubt that is the case. Why doesn’t it show up in video encoding? Well that’s something important for Apple to investigate.
You aren’t just comparing hardware. You are comparing the Operating System and it’s standard libraries. You are also comparing the quality of the application port.
Fundamentally, the Apple OSs (iOS, macOS, tvOS, iPadOS, watchOS, bridgeOS, etc.) have a serious following. macOS, for example, since its initial releases is an OS that you just start using. You don’t need stack-overflow, google or anything else. macOS is also the slowest of the 3 main OSs on the desktop market, by a considerable margin. All benchmarks in the recent years on identical hardware have shown Darwin to be considerably slower than Linux and Windows. I find it to be fast enough, but not spectacular.
OTOH, the hardware comparison of the A-series chips with their Intel counterparts leads to impressive results. Since you have the same OS on both sides (macOS and iOS), and the same application (e.g.: Geekbench), the performance measurements directly compare an Intel CPU and an Apple ARM CPU. An Apple A12X Bionic @ 2.5 GHz has a single core performance of 1112 units, which is inline with Intel’s fastest CPUs and a multicore performance of 4603 which is inline with 4 and even 6 core Intel macs. Extrapolating that from the 5-10W TDP of the A12X, it would mean that a 35W SKU as used in a MacPro would get a 16000-20000 multicore score (by increasing cores, not frequency), which is considerably faster than the current top of the line Intel Xeon W-2191B @ 2.3 GHz (18 cores) found in the iMac Pro which can barely squeeze 13000 units of score.
I personally am quite satisfied with the incremental updates of all Darwin branches, especially when I have the misfortune to meet with a Windows 10 PC. And I am quite excited about the performance of the Apple ARM64 chips and their potential in upcoming laptops and desktops. Architecture wise, IA32/EM64T/AMD64 are showing clear limitations and it is quite evident that ARM64 still has some headroom to grow, even considering the major slowing in lithographic shrinking. At the same scale and power budget, an Apple designed ARM64 can clearly beat a competing Intel/AMD by at least a factor of 2 and that opens up possibilities.
If you look at the Intel SKU portfolio, it is clear that they are doing their best to make money on shit products. The Xeon CPU line appeared in the Klamath era and it costed $2000 for a Slot2 CPU with at most 6 SKUs. Right now it has a few hundred SKUs, with little real difference between them (there are probably 10 SKUs with 8 cores at 2,5GHz), while costing anywhere from $400 to $13000 per CPU. Same goes for the desktop CPU market, you can find anything from dual core CPUs (in 2019!!!), to 28 core CPUs with a TDP of anywhere between 1W/(GHz*Core) to 5W/(GHz*Core).
The author of the article is comparing the speed of hardware assisted H.264 encoding for 4k@60p. It’s fine, but it’s not comparing the quality and bitrate performance. If it’s 10% faster but almost half as efficient at compressing (since it’s really a compressing benchmark), it’s not really called winning. The criticism surrounding the iOS file management API limitations is partly justified. But it also means that iOS has been an incredibly secure OS by comparison in the past years. The fact that from moment zero, all iOS apps have been sandboxed has made file management more difficult, but also security better. It was a tradeoff. For the author it was a bad one. For me, it’s a good one.
Let’s face it, all mobile operating systems are troubled, incomplete and buggy, and the vendors have a difficult job of balancing the needs for fixes, improving in software architecture (which don’t have benefits outside the development teams) and adding new features. Imagine that you are responsible for software as old as Adobe Acrobat. You have to push a new version every year with new features, but the code is old and incredibly difficult to maintain. It supported Windows 3.1, Linux, NextSTEP, UnixWare, Solaris, SGi IRIX, Windows and many other platforms. It would take 1 year to remove all the cruft and leave it with just support for modern Windows, modern macOS/iOS and modern Android. Do you think your customers would buy the upgraded version if it gave them nothing in return? Sure, it should be a bit faster, but not enough to justify spending $300 for the upgrade.
> Let’s face it, all mobile operating systems are troubled, incomplete and buggy, and the vendors have a difficult job of balancing the needs for fixes, improving in software architecture (which don’t have benefits outside the development teams) and adding new features
Starting from Unix derivatives both on Android and iOS for what used to be the dominion of realtime OSes was not exactly a great idea. It looks like in either case, they used what was the fastest to throw into the market, not the smartest for long term thinking. Now Google is preparing its Fuschia OS to repair their technical failures and Apple is… probably not doing anything smart, software-wise, as usual.
Spot on. Blackberry tried to do it right with QNX, as you can see the market didn’t honor it.
The Kernel is one thing, the App Store is another. Provided you can hide the Kernel’s misbehavior well enough, the general audience doesn’t care. It’s about usability, community and price.
Blackberry did not “try to do it right,” QNX was literally the only OS vendor they could afford to acquire, because their own Blackberry OS was a dead end and sucked balls. They got QNX not for the “real time” aspects of it, but for the unix/embedded angle.
Turned out that QNX was also a shit idea because it had no ecosystem on that application space, and it would have been easier for Blackberry to go android from the get go… which is what they ended up doing in the end (but it was too late).
iOS and Android work just fine, unix has proven to be a rather scalable design. Seriously, its hilarious to read these comments by the armchair quarterbacks. At the end of the day, companies have to ship products not conduct “purity” rituals seeking the perfect OS (which does not exist) the design of which will please the couple of random geeks who don’t know much at the end of the day.
I don’t think many of you understand what it takes to ship a large scale software project like iOS or Android.
UNIX-derived systems do just fine in RT work (see MINIX 3 or QNX for solid examples). The issue here is that neither iOS nor Android were designed for RT usage for anything other than the phone aspect (which they don’t even need to do much with, as almost all of the stuff that needs to be RT gets offloaded to the baseband processor).
Exactly, for the most part the main OS for a cellphone does not need any RT facilities.
Perhaps some people in this thread do not understand that a modern smart phone has several OSs running. The baseband subsystem runs its own deeply embedded OS, for example. A modern phone/tablet really are many different computer domains, Android/iOS being the only user facing ones.
d3vi1,
What subsystems specifically are slower?
ARM is making nice gains. I’d say it’s a bit contrived to compare mobile CPUs to desktop CPUs since they’re optimized for different environments, but to the extent that some of the same benchmarks are available for both, it’s interesting to compare them.
https://browser.geekbench.com/ios-benchmarks
https://browser.geekbench.com/mac-benchmarks
https://browser.geekbench.com/processor-benchmarks
ARM processors are making good gains, IOS on ARM fairs well against OSX on intel, but this is in part because apple macs don’t use the fastest intel processors available. When you look at intel processors overall, they still have a ways to catch up.
The fastest IOS scores: Apple A12X Bionic @ 2.5 GHz ST=1112 MT=4603
Here’s the score from a system I built last year.
https://browser.geekbench.com/v5/cpu/138649
Intel Core i9-9900K ST=1408 MT=9415
My desktop is 26.6% faster on single threaded loads and 104.5% faster on multithreaded loads. Of course it’s not exactly fair to compare desktop and mobile performance, those desktop CPUs use a lot more power. One thing I will say is that at this point nearly all users have more than enough for their needs for both desktop and mobile. Outside of niche use cases, it’s rare for users to use CPU to their full potential.
Dude, you’re comparing an overclockable i9 to an ARM cpu. They aren’t there yet. Try comparing an i3 or i5 to a the ARM.
Bill Shooter of Bul,
Haha, yes I thought I was clear that it’s not fair, I agree with you. However it was in response to d3vi1’s post where he said apple’s ARM processors are “inline with Intel’s fastest CPUs”, hence the comparison.
Incidentally my CPU is stock, not overclocked and I’m not too interested in trying it myself. Overclocking is relative anyways, with the intel 9th gen i9 cores, I don’t think intel actually made any process improvements over previous generation. I suspect it’s mostly just better binning of parts that had more overclocking headroom combined with a soldered lid.
If you look at those links you posted for single core benchmarks, it is clear that we are close to the point that all low hanging fruits were already taken for int and float operations (SIMD being a special case) and most of the improvements came from faster clocks so, in my opinion, there is nothing much to brag about it now by ARM camp. For multi-core, the situation is very different. Granted, the typical usage pattern of a phone is one task at time, so it is not of surprise that Apple processors seems to be doing so well on tests.
What I really would like to see is how well ARM processors behave under intensive IO on multi-threaded environments. There, we don’t stop everything else to test very specific things, and how well things go on synthetic benchmarks does not mean it would do wonders under what we find on many computing scenarios.
I remember seeing a comparison done many years ago between Intel and IBM hardware under business typical tasks loads (DB and stuff like that) and, even though Intel was way faster on int and float operations, it was way behind on latency when simulating huge number of parallel connections.
Of course, for regular users you are right to point out that we are close to the situation where modern processors are already good enough. For really professional things like rendering of movies in 4/8k, simultaneous access of many users and other things where IO and parallelism dominates, things can be a lot different, and modern PC hardware is build to do well on this scenario (perhaps, because of games).
Last, power usage is a very difficult thing to talk about, specially because it is non-linear, and Intel hardware performs very well under less extreme clock rates but, still, as it was not one of their main concerns, not quite as good as on ARM camp.
acobar,
Yeah, I’ve tried several ARM CPUs over the years and the last purchase (Odroid N2) is the first I’ve really been impressed with about the CPU performance. If you have any benchmarks in mind, I can probably run it. However the driver situation is so bad that I cannot recommend it if you plan on using the desktop. It’s much slower than an accelerated desktop on a slower CPU. Unfortunately bad/unsupported drivers are a reoccurring theme with ARM SBCs
I take geekbench scores at face value, but I know there’s a risk in doing so. I wouldn’t mind trying out apple’s ARM processors for myself, but it’s less useful to me in a phone/tablet factor not to mention IOS restrictions disqualify them for anything I need them for.
Clock rates scale quite linearly assuming you can keep the power/temps in check. In my experience the benchmarks and clock speeds are perfectly correlated between comparable product families. That can be useful because if you don’t have benchmarks for a specific processor, you can probably extrapolate it. However power consumption grows exponentially, as you say.
I’d really like for ARM to evolve beyond mobile and into intel’s workstation territory (ie with more metal and muscle) but there aren’t many ARM vendors offering anything like this at the moment.
Exactly my point. Apple can produce a CPU that is at 78% of the single thread performance and 48% of the multi-thread performance of a CPU that used 10-20 times more power and proportionally produces 10-20 times the heat.
For Apple it is easy to scale a 2,65GHz core to 3,34GHz to match the single thread performance and to double the cores. Let’s do a bit of math on this: the speed increase would be (126%^2)*5W=8,01378W and the core increase would be 8W*2=16W.
How does your 95W Intel Core i9 9900K compare to a hypothetical 16W ARM64 from Apple having the same performance values?
Let’s do this math the other way: an iPad SOC (A12X) has a 5W TDP. The fastest intel CPU at 5W is the Intel Core i7-8500Y. The scores for it are 717 for single core and 1235 for multi-core. Compare that to the 1112 single core and 4603 score of the A12X. Apple is 55% faster in single-core and 272% faster in multi-core in the same power and thermal budget.
Sure, they haven’t yet released ANY desktop class CPUs, but 272% faster than Intel is not a small feat. The current top of the line MacBook CPU is the Intel Core i9-9980HK. Now imagine something that is 55% faster in single thread (roughly equivalent to 3,72GHz base frequency) and 272% faster in multi-thread (basically the same as a 30-core 2,4GHz Core i9). Even if you include the exponential TDP Frequency scaling, you will have room for 16 cores in the same TDP budget and you get the neural engine, the GPU (that also scales impressively). The GPU in the iPhone X/XS/Pro already can render games better than the MBP at 65% of the resolution of the MBP. The MBP display is at 2880×1800 and the iPhone11Pro Max has 2688×1242.
d3vi1,
Your “It’s easy” part trivializes things that historically not so easy. The first megahertz and gigahertz are always the easiest, but the difficulty and power consumption and probably even costs grow exponentially. Also, another point is that adding more cores usually doesn’t scale linearly due to additional overhead in the multicore fabric. For example, in typical applications, at least those that aren’t embarrassingly parallel, you’ll find that 8cores x 4GHZ is not equivalent to having 16cores x 2GHZ. So an SMP workload that works well on 8 faster x86 cores could actually perform poorly on 16+ slower ARM cores dispute having a theoretical raw computational advantage.
In principal, there’s no reason ARM couldn’t match x86 performance, but apple is clearly not there yet. Still, I’d really like to test apple’s CPU in an unlocked SBC form factor, but chances are they won’t be releasing it outside the confines of IOS.
Same criticism here, you are basing your math on linear extrapolations that don’t necessarily represent the actual difficulties of scaling in practice. It’s well known that intel processors use more energy than ARM counterparts, however it’s not at all obvious to me that an ARM processor targeting the same performance profiles would do any better. One way or another I would love to see it, intel needs competition especially on price, but I’m not really expecting any miracles.
I said this in a previous post, but the top of the line macbook does not use top of the line intel CPUs. In fact even without regards to comparing x86 and ARM processors, all of the recent products in apple’s macbook lineup are notoriously bad for performance problems due to apple sacrificing performance for aesthetics (look it up). My first time using a 2017 macpro came as a disappointment to me when I did some hard number crunching on it and pretty much encountered the same bottlenecks. Anyways I still think apple’s progress on ARM is impressive, but it’s still way to early to write off x86 as the clear leader of high performance workstations.
Wait a minute. From the summary:
OnePlus also delivers a final video at twice the bitrate of the iPhone (which does look better to my eye). Better quality, twice the size, in two thirds the time.
So… this is the person complaining about synthetic benchmarks? When this person is using very *different* encoding parameters to give their preferred phone a so-called edge?
Clickbate!
Yeah his bitrate understanding is flawed. But you are complaining about the first test. Later in the article he compiles the same video using the same app and the same settings on both platforms, and the iPhone loses.
Then there is the matter of the XR losing to the SE with the same settings and app.
It’s unfortunate that he leads with a test that is basically meaningless, but there are others that are better and none of them show the iPhone in a good light
darknexus,
Yeah, I think he misunderstands the tech a bit. You could compress to standard MPEG or MPEG2 much faster and with much higher bitrates, but that doesn’t really mean anything since it’s less work than a modern codec. The codec need to be set to the same settings in order to compare benchmarks fairly (although sometimes software makes this impossible because they don’t expose the codec settings).
This article is a load of bullocks: it compares completely different encoders which produce completely different output and makes outrageous conclusions. You want to make an apple-to-apples comparison? Compile the same encoder, i.e. x265 for both platforms and run it for the same source file using the same encoding settings.
The A13 SoC is miles faster than the fastest android chipset, SnapDragon 855+.
See GeekBench 5 results and see for yourself. If you have an aversion towards GB5 there are multiple other benchmarks.
birdie,
To benchmark the hardware itself as the only variable, sure that’s the way to go. However it’s pretty clear the author was testing different software from different vendors, it must be stressed that this is a legitimate thing to do! With video compression we are frequently faced with A & B testing and have to decide which is better. It would even be legitimate to compare completely different codecs with different settings if he so wanted. However IMHO he makes the crucial mistake of assuming higher bitrate is better and/or equates to more work being done.
His understanding of bitrates is backwards. There are just too many variables here, but clearly all else being equal, a lower bit rate is better! If I can get the same quality with half the bit rate, even if it takes twice as much work, then it may well be worth it. This is the whole point of compression.
With this in mind, his conclusions about using the software with their default settings may still hold up. Most users will use the default settings, and I don’t doubt that IOS video software could be slower using default settings. If encoder time using default settings is all you care about, then you can stop there and his conclusions are sound. However his whole gripe with encoder performance glosses over the output product. He’s failed to make a convincing case that the IOS results were worse, meanwhile the lower bitrate output by IOS (by half as much!) is technically a big win for the IOS software even if it took longer.
I do the same thing in going to benchmarks like this, but we also know that benchmark cheating is a thing that happens. Vendors that want a better score can overclock, disable GPU features, etc to get an undeserved rating boost and there are vendors that have been caught over the years. Who knows how many haven’t been caught? Hell even some car manufacturers have been motivated to cheat on benchmarks. I’m not really sure if apple would do it (they had no problem deceiving their consumers about updates that artificially induced performance loss on older models), but it’s something we need to be aware of.
Not in my universe. Not when you’re claiming that based on these oranges-to-apples benchmarks that the Apple A13 SoC is slower than Android chips.
In short your comment is without substance.
birdie,
Do you live in a universe where choices are already made for you? Because A/B testing is kind of important to compare these different yet similar products from multiple vendors.
Whether you acknowledge it or not is irrelevant, it’s just a fact of life when different vendors sell similar products that consumes and professionals alike are forced to chose between the relative merits based on the real or perceived cons & pros of each. A/B testing can be scientifically rigorous way to compare quite different technologies.
Look, I know the author did not do scientifically rigorous testing and I criticize him for that, but you are wrong that A/B testing different products could not (or should not) be done in principal. Every time a new codec comes up, do we throw up are hands and say we cannot compare them because they don’t and can’t produce identical bitstreams? No! We compare the hell out of them through A/B testing! When done right, it IS a legitimate thing to do!
There’s cables you can copy data off the phone with so you don’t have to attempt to process it on the phone.