The Hitchhiker's Guide to Encoding: And Another Test...(Or PSNR and all that...)

Hi Andy
Can you tell me if the BBC sujectively compare BBC HD vs competitor HD (Sky for example) in the same way they compare new encoders vs old encoders?
To be honest, although your blog this week has been interesting, I still don't know why Sky HD looks much better than BBC HD.
I really don't care what bit rate / encoder combination you use, either now or pre Aug 09. All I want is a BBC HD service which is at least as good as Sky HD.
Bob

Everyone from 'Geeks' (to quote Freesat, not any BBC staff) to my wife, to shops and 99% of people on these posts have seen a big reduction in picture quality in the last year. Why don't you try to explain this and answer the seven questions, rather than join the great Freeview HD cover up. At what point in you week of blogs are you going to try to answer the questions being asked.

Another day and still none of the questions answered.


It has been a fantastic blog Andy, and a very very interesting read. We are still however left with some very easy questions that have not been answered.


I await tomorrows blog with interest, not because I am expecting answers, just because i find them interesting.

Lee.

I'm beginning to lose the will to live here.....

What is the point of all this?

I watched Robson Green's Wild Swimming on ITV HD. The PQ was absolutely streets ahead of BBC HD. You should be surpassing that standard. Why aren't you?

Be warned. Not long to go now on this BBC PR exercise. Trust me, it's not working!

Andy,

If I'm correct in thinking what you are saying is the new encoders are showing up noise more because they are more sensitive right?

If I bought something and it wasn't upto the job it was designed for then I'd send it back and go somewhere else!!!

All this is very interesting but it STILL doesn't answer the questions!

I agree it's all interesting stuff - however all it's saying is the usual 'there is nothing wrong' statement, yet hundreds of viewers are saying the opposite - so who is right? Will the BBC move on this? Why does Andy refuse to answer the questions asked - those 7 fundamental questions we all want answering?

The blogs so far do not answer any of them and whilst the blogs are an interesting read I am sure we would all prefer to read definitive answers to the questions that matter.

All we are getting is smoke and mirrors.

Thanks once again Andy - an excellent post.

Your final comments re picture noise in dark scenes were particularly interesting and pose and interesting question to me.

It seems that someone somewhere will have to make a decision about applying noise reduction to 'nuke' camera noise. This as far as I can tell, could lead to a situation of arbitrary noise reduction even when a grainy look is intended. I for one would not like to see a compromise of the artistic intent of the programme-makers. This comes back to my earlier points about transparency of encoding - ie that the final encode should represent the source material - for good or for ill.

It seems to me that if we could rely on the encoder being transparent, then should complaints arise regards dirty, noisy images, then these complaints could go to the programme makers directly. Changes in film-making trends and styles would be more readily dictated by public feedback.

With regards to grainy images, the difficult part of all this is a lack of willingness to differentiate between something one doesn't like and something that is wrong. I for one had an almost off-topic discussion about film grain with @alsone a while back and it is clear that this subjectivity of taste can seriously muddy the waters!


Also, I must say that the comment about the piture-tester seeing blocking in the passing scenery made laugh (LOL as they say on t'internet). Having had a particularly busy week in front of Photoshop, I found myself looking for the 'Undo' on the oven recently.

Interesting stuff again Andy, even if it did require 2 cups of coffee on my part!

But do you not think that the first graph comparing old v. new encoder DOES, to quite a large degree, actually bear out what many posters on here perceive about picture quality on BBC HD since the encoder change?

For instance, I think most on here have acknowledged that when the quality of source material is high e.g. Emma, Dr.Who, selected sequences on Life, then PQ still looks excellent, but perhaps with just that real 5% extra 'wow' factor not quite there as there was previously. For example comparing Emma to Tess Of The D'Urbervilles (shown SEPT 2008), both similar productions in style and shot on 35mm (IIRC), TOTD is probaly the best PQ I have seen in 3 years of having HD, but I wonder if it would have looked quite as good if shown now?

Conversely, on something like 'Garrow's Law' where there were a lot of dark 'noisy' scenes there were many instances of very bad 'blocking' that were easily visible to the naked eye, that I do not recall seeing in similar style productions in earlier times. Again your graph actually does seem to suggest this is the case.

So although in percentage terms it may look mathematically as though the new encoders are better than the old (regardless of bitrates), the dificiencies are visible at just the times when you don't want them to be (see above)?

So when you add in the fact that BBC HD as a SINGLE channels broadcasts a wider of range programmes from varying sources than any other UK HD Channel I can understand why it can be such a headache delivering high quality with the fixed resources that the BBC has. Also, I don't think any HD channel in the UK would stand up perfectly to the level of scrutiny that BBC HD has recently - for instance I have seen bad cases of blocking on Sky 1 HD e.g '24' and the Movie Channels e.g. The Bourne Ultimatum.

But everybody on here has taken up HD at relatively early stage in the UK, because we enjoy stunning picture quality - and we all want BBC HD to be the best.

Anyway - look forward to tomorrows blog.

Thanks.

Andy, have you used structural similarity (SSIM) for comparing images?

Quoting from wikipedia:

SSIM is designed to improve on traditional methods like peak signal-to-noise ratio (PSNR) and mean squared error (MSE), which have proved to be inconsistent with human eye perception.

I would definitely like to see the results of that test.

So are the graphs showing the PSNR of the old encoder running at 16 Mbps against new encoder running at 9.7 Mbps? And of which test sequences? Is there a PSNR of test sequence with high quality motion eg. 50i and 50p (none of this low 25p stuff) like a whole 50i live TV programme with the old encoder at 16 Mbps vs the new one at 9.7?

"In accordance with industry recommendations, only the luminance PSNR is measured"

Doesn't this mean the encoder could throw out all the colour information and show only black and white video and have an excellent PSNR? If we are watching colour TV shouldn't colour be taken into account?

Also, was the testing done at screen size to viewing distances that are recommended for standard definition viewing or those recommended to be able to see full 1920x1080 resolution high definition viewing. And why does the BBC continue to output 1440x1080 when newer cameras/other sources can capture 1920x1080 or higher?

Are the test results (eg. by viewers) for the old at 16 mbps against 9.7 mbps new encoders available anywhere to see, and how the tests were done? Will they ever be redone at the correct screen size to viewing distance to be able to resolve the full HD resolution? Will the BBC ever broadcast the full HD resolution? Will we be able to see see the test results for each of the TV types? Were all the TV types in the test full HD ones? Won't Plasma and especially CRT TVs hide artefacts a lot more than an LCD TV?

How did the encoders compare with the EBUs 1080p50 test sequences?

Andy,

I would have liked to start my comment on this blog with the usual polite thank you, but I can't, so I must apologise in advance for the tone of this post, fuelled by sheer frustration.

I will be blunt. You have let us down. The only comparative results pre & post encoder change/bit rate reduction that you have given us are luminance PSNR. If that was the only quantitative test done for PQ (and I don't believe it was) then it is no wonder the BBC's testing didn't predict the degradation in PQ we are seeing. It would be very easy to reach the conclusion that of all the parameters tested, these particular ones were selected for release to us because they are the least damaging to the BBC's technical standing when put in the public domain.

I think you have grossly underestimated the intelligence of the people who post on this and the predecessor blogs and I certainly don't appreciate being patronised in this way.

When I reached the heading "Subjective expert viewer evaluation" my spirits were temporarily raised, as these in comparison form would surely demonstrate the picture degradation that is there for all to see. But no, all you have done under that heading is to described the testing methodology. Where are the results of those subjective tests?

Oh, and in case you might have missed them, these questions still remain unanswered:

1. Why was the higher bit-rate not sustainable?

2. Why can't we have a short period back on the higher bit-rate as a simple test?

3. Why can't the satellite transmissions be statistically multiplexed to make better use of the available bandwidth on the transponder?

4. Why is BBC HD being transmitted in Scandinavia at a higher resolution and a higher bit-rate than that being transmitted to the licence paying public in the UK?

5. Will the BBC be increasing the bitrate in the near future? Yes?No

6. Will the BBC be changing the picture format to 1920x1020? Yes/No

7. Will the BBC be introducing a variable bitrate? Yes/No

Only 1 opportunity left Andy.

Andy,

You talk about the PSNR of the football match with the mix fade issue:-

"While we had the location recordings of the match to analyse the mix error, we had a chance to compare the PSNR curves through the new and old encoders"
"For the majority of the sequence, the new encoder has a higher PSNR than the old with a median increase of about 0.2 dB (not a noticeable difference)."

How can you reconcile the PSNR median result of 0.2 dB (not noticeable) with the very obviously visible problems caused by the mix/fade problem?

This type of analysis is obviously flawed, I will correct that, the interpretation of the results is obviously flawed.

I don't care if a high percentage of the material isn't demanding enough to give a similar PSNR to the old encoder, I care about smaller percentage that makes the encoder dissolve into a blocky mess.

Also, if a PSNR of 50db is ideal, and 30-40db are typical, what figure is BBCHD actually achieving?

You also forget many found the previous encoder inadequate. Is it a case that all you have done is match an inadequate standard?

Not that IMHO you have matched it on certain demanding material.

This all looks like smoke and mirrors to me.
Andy can go on all he likes about the new encoder v/s the old encoder, but in the final analysis, the BBCHD picture quality has deteriorated markedly, and if Andy and all the other high-ups cannot see this, I despair.
Probably they can see it. But they are denying it for their own internal reasons, and probably hope we will all go away.
I too would like an answer to the seven unanswered questions:

1. Why was the higher bit-rate not sustainable?
2. Why can't we have a short period back on the higher bit-rate as a simple test?
3. Why can't the satellite transmissions be statistically multiplexed to make better use of the available bandwidth on the transponder?
4. Why is BBC HD being transmitted in Scandinavia at a higher resolution and a higher bit-rate than that being transmitted to the licence paying public in the UK?
5. Will the BBC be increasing the bitrate in the near future? Yes?No
6. Will the BBC be changing the picture format to 1920x1020? Yes/No
7. Will the BBC be introducing a variable bitrate? Yes/No

Andy

Thanks very much for this interesting insight into the testing of encoders by way of the mathematics behind what is deemed to be an improvement of PQ due to new hardware and software implementations.

However, your blog does raise a few interesting points. Firstly, in the same way I would think that new digital mixes of existing analgue masters for audo CDs brought to life a whole host of noise and previously unknown background errors on remastered albums, so the new encoders, by the looks of it, will show up to us some noise on the broadcast that might not have been there before. Fine - no problem with this as, if this was the only perceived issue by us all, then as the new encoders would be showing up information present in the broadcast that wasn't previously there, but the overall quality of the images seen was maintained, then I honestly don't think people would be too worried by this as it would be able to be managed, as you have inferred in your post, going forward.

Sadly, this is not the case. Regardless of the maths and the PSNR result, this information can be, and I would bet my bottom dollar on this in subsequent posts by others more akin and in the know about these type of results, will be interpreted in a different light to your very thorough explanation above. As such, we will be in a position of your interpretation of results implying one thing and someone else's another.

However, surely the consistent comments from all about the degraded PQ is what matters. Previously, for example, the background lighting on Jules Holland was pin sharp, like looking through a window on the set itself - pretty much as we all expected HD to be. Nowadays, the areas between dark and light are blocky which never used to be the case. The maths won't show this is what my money is on - the human eye will and does, and that's not from being sat in a darkened room looking at this for days and weeks on end. This is observation from being sat less than seven feet from a good 46" LCD.

Something has gone wrong somewhere - surely you must agree that it has to have had such a response to this issue (hello Independent). Unfortunately, if simple questions to simple questions posed are not answered, there will be a backlash like nothing seen on these blogs before.

I applaud your blogs Andy as they are very interesting and have offered a good insight to what goes on with the testing etc. I hope the concluding part tomorrow gives us all the answers to the questions that we're after.

Roll on Friday...

"The nice thing about standards is there are so many to choose from"

Well the nice thing about metrics is there are so many to choose from. This article is quite interesting as it contains photographs of what PSNR looks like in a frame. It also discusses some other metrics.

https://www.compression.ru/video/quality_measure/info_en.html

The first thing to say is that PSNR does NOT have a high correlation to subjective picture quality (eyeballs). In fact it is one of the worst metrics as it takes no account of the non linearality of the Human Visual System as is shown in the formula. There are metrics which do correlate well but they do take into account the HVS. This does not mean that PSNR is useless. It is usefull in makeing certain comparisons and there are ways to calibrate the PSNR against visual perception. All this is not made easier by the fact that visual perception is a very complicated subject.

I too am very concerned that only the luminance has been measured. It means that the encoder could be outputing gross colour errors and the PSNR would not reflect it. I am also concerned about the use of 100mb/s mpeg2 all be it I frame. As you know there are interactions in successive encoding. I would have thought it would be fairly easy to find some uncompressed clips.

Quote: "..on the train just looking at the country side passing by when he was convinced he saw compression blocking in the leaves of trees..."

I really did laugh out loud! A great story.

BTW, Has anyone compared the GOP contents from each encoder?

"..on the train just looking at the country side passing by when he was convinced he saw compression blocking in the leaves of trees..."


Mother nature cut the bitrate by 40%, but all is fine the new encoders are working without issue!

(and yeah I chuckled at that story too)

@6 hd blogger

'If I'm correct in thinking what you are saying is the new encoders are showing up noise more because they are more sensitive right?'

I would say what we want of a transmission chain is absolute transparency. If there is picture noise on the source then this should be faithfully reproduced.

Unfortunately, as someone else has said, that requires more bits to encode and therefore less for the 'normal' picture content. That, I suggest, is one reason for visible artefacts. This can only be remedied by having more bits available - even if only momentarily.

That is why the subjective quality of the Sky HD channels is the equivalent of 14 or 18 Mbps. They employ statistical multiplexing and can share around the overall transponder bandwidth between several channels according to need. Statistically it is unlikely all channels are attempting to encode difficult material at the same instant so it makes for a very efficient use of bandwidth while ensuring good quality for 99.9% or the time.

I'm sure all the Boards of the banks had figures to approve their toxic loans and that the Fees Office in the House of Commons had their lists of approvable expenditure. It's just a shame that the real world got in the way of their cosy tests.

I'd rather have my noisy images shown as noisy images as they are elsewhere and not the blocky mess that these 'better' encoders produce.

Can the new encoder in its current state just not be good enough to meet expectations?
The only way to get the old PQ back is to go back to the old encoder at the old bitrate.
It hasn't been been shown that increasing the bitrate with the current encoder to 16 Mbps will improve anything. One would hope they did this during testing and found the current bitrate sufficient 'for the current capabilities of the new encoder'.
Any future improvements will need to come from software upgrades (which may or may not be able to take advantage of increased bandwidth, fixed or statmux'd).
That is the conclusion I am expecting tomorrow.

Well done Andy another very interesting blog with lots of information enclosed in it. But as others have said and what the majority of viewers want to know is why you are broadcasting at the level you are and how the BBC feel it is acceptable to broadcast a 2nd rate service and give the license fee payers what we deserve

@20
"I would say what we want of a transmission chain is absolute transparency. If there is picture noise on the source then this should be faithfully reproduced."

I agree!

The problem is, if you choose to pre-filter the signal to remove noise, you also remove (a) the fine details in the picture and (b) any 'artistic' grain effects (admittedly, frowned on by the BBC anyway).

We have a fundamental problem here: If the signal is filtered to remove noise, the results are heralded by us as 'not looking HD'. If it's not, the encoder does nasty things which we can all see and which we then complain about.

@13
"How can you reconcile the PSNR median result of 0.2 dB (not noticeable) with the very obviously visible problems caused by the mix/fade problem?"

Again, spot-on analysis!

I would have expected the PSNR to have shown huge, gross errors. I suspect the published graph does not include any scenes where racking changes produced blocking effects. Is that correct, Andy?

I'm dying to see the new fix for this issue Andy alludes to. I hope it's better than the 'temporary fix', which doesn't seem to work too well.

Andy, when do you anticipate getting this new software, and is it more than simply the 'temporary fix' converted into code?

As to the basic point of how it can be possible that we all see the apparently near-invisible, if not non-existant: I just don't know. I do know I trust my eyes and judgement, and I shall be continuing to call attention to the shortcomings of BBC HD's output for as long as it takes!

You know Google auto-completes search entries? Based (I assume) on the popularity of the search terms.

Well, if you type BBCHD into Google, the fifth most common next word is "bitrate" - and if you start to type bitrate, it auto completes with "BBC HD bitrate reduction" and "BBC HD bitrate cut".

Seems a lot of people want to know about this!


I'm far more interested to know what the space cleared by reducing the BBC HD bitrate is going to be used for.

At the moment, it's empty. It's strange to broadcast nothing. Even if the results at 9Mbps were "good enough", given the amount of spare space you could be transmitting an even better picture than before by using all that space.

[b]So what has the space been cleared for?[/b]

Cheers,
David.

A second HD channel? World Cup football in HD? Maybe they will be trying to squeeze two HD channels into the space now vacant because of the bit rate reduction. Only they can tell us why the old bit rate was 'unsustainable'

Thanks Andy,

This is the most interesting post so far this week! Unfortunately, it seems that the results of the PSNR tests can be interpreted to say what you like. The link posted by tjharris #17 listed other tests which would appear better suited to highlighting the issues with the new encoder that your viewers are experiencing. Any plans to use those or similar tests?

The conclusion today seems to be the new encoder does some stuff better and some stuff worse. The non-expert-viewers notice the worse stuff more than the PSNR tests reveal.

On one hand I'm sad to see that the mass-media has now got hold of this story as they like any excuse for Beeb bashing. However, this has been brewing for some time, and the bad publicity could have been avoided by listening to your viewers when the complaints started coming in several months ago. I feel there has been a certain amount of arrogance displayed by the BBC HD management, and not listening or acting has resulted in the negative publicity campaigns we are now starting to see. I think the BBC license fee is great value, but there's no point in running an HD channel if it ain't HD...whatever the graphs say.

The Daily Mail has the story now:
https://www.dailymail.co.uk/news/article-1234797/BBC-viewers-quality-HD-channels.html

Thanks to Mike Porter on the prvious blog for the heads up, off to read it now...

Oh dear - the Daily Mail now on board. That is not good for the BBC hierarchy I'm afriad and could have been easily avoided.

If the questions are just answered and people honest and accepting of what we license fee payers are reporting, then what is looking like turning into a bit of a sorry mess need not have happened.

Andy's blogs are very informative and I am awaiting tomorrow's conclusion with something approaching bated breath (alright, so I've got the day off work!), but now that 2 papers are reporting it, MediaGuardian can't be too far off...

I have now had time to digest the two graphs that Andy has given us. I am dissapointed with how little information he has given us but I think I can explain why the graphs look so good but our eyes tell us how bad things are. The tool I use for measuring PSNR does not present the graph like this so I have had to do some thinking.

First I am assuming that right hand limit of the graph is 50db and that the graduations are in decibels. We are not told if this is for one clip or if it is an average over all the test sequences but this does not matter at the moment.

The first thing to notice is the very wide range of PSNR the frames have. The worst frames are 31db and the best frames almost perfect at 50db. 20% of the frames have a PSNR of less than or equal to 34db. That is awfull and explains why the new encoder looks so bad but the question is why did the old encoder look better. Well we need more information to explain that.

These are the PSNR for the Green Valley encoder for two of the EBU clips.

Croud Run @ 10mb/s 29db
Croud Run @ 16mb/s 31db
Flower Close @ 10mb/s 39db
Flower Close @ 16mb/s 41db

This shows us there is a 10db difference in PSNR between the two clips. This explains why some programmes of parts of programmes can look quite good. The quality is very dependent on the complexity of the clip. 10db is a massive difference which anybody would notice.

You will notice there is only a relativly small difference of 2db between 10mb/s and 16mb/s, small but perceivable by most people. But this is not the whole story. It is not the average PSNR which is importent it is the minimum PSNR. Viewers are complaining about the complex parts of the program where the PSNR drops to unacceptable levels.

This all shows the how much better it is to use a variable bitrate. Many HD channels using variable bitrate peaking at 19mb/s which will increase the PSNR for complex frames.

I would like Andy to publish the same graph with the new encoder at 19mb/s as I suspect we would see a much higher minimum PSNR.

I suspect that what I have said here is not the whole story as the old encoder did look so much better. We must remember that PSNR does not have a good correlation to perceived picture quality.

As mentioned earlier all these blogs are just smoke and mirrors and basically a waste of time. Why don't BBC HD just flick a switch and up the bit rate and see what happens?

It's sad to see the channel with so much bad press but BBC Management have only themselves to blame - Danielle Nagler has head of the channel needs to take FULL responsibility for the state the channel is in now. I thought she had a good rapport on the blogs with the viewers - yet she seems to have gone to ground.

Andy - sorry to say this but you too are losing credibility with many of us - I understand you have to stick with the BBC's stance but surely professionally deep in your mind isn't there a niggle that something isn't quite right?

I'm surprised no-one has e-mailed The Sun yet.

Sorry, but this is a most utterly ridiculous set of blogs.

I don't care about the science of encoding. Nor do I expect 99% of the other people who have complained about the poor quality of BBC HD.

All I and they care about is the end result. When are you going to address THAT, and answer the questions which are still outstanding?

Pointless obfuscation in an attempt to blind us with science.
However, it doesn't make me blind to the obvious inadequacies of BBC HD...

This comment was removed because the moderators found it broke the house rules. Explain.

I agree with others in that it is not clear what is being tested here.

Are the results for the "new encoder" also at the new lower bitrate?

I assume you tried the new encoder at the old bitrate? What difference did this show?

How did you decide on the new bitrate? Surely tests were carried out at many different bitrates to determine which would be used? Why was the chosen one chosen?

And I've assumed the "old encoder" also means old bit-rate but I guess it might not! Can you clarify please exactly what bitrates were used in the graphs?

"He said was on the train just looking at the country side passing by when he was convinced he saw compression blocking in the leaves of trees."

Ive seen that too! Honestly! Now where did I leave my straight jacket lol :)

nur0 comment #28 -

Looks like Daily Mail almost took my post word for word :)

Dave K.

Another scenario to lighten the mood:

Manager:

I'm in a heap of trouble with this issue. The board don't like the negative press we are receiving and the accountants are tightening the purse strings all the time. I would like you to personally sort this for me and get me off the hook. Consider it an additional objective on your annual appraisal

Engineer:

It's going to be difficult. They are a clever bunch out there. They will see through any technical arguments if am not completely honest with them.

Manager:

You work for us, not them. Put your professional pride to one side until this whole thing blows over. They'll get tired and go away eventually. How about producing a series of deeply technical reports. Spread them out over an extended period of time, and keep refusing to answer the difficult questions by giving them the impression the answers will be found in the later reports.

Engineer:

I don't like it but it looks like you aren't giving me any option, so I'll give it a try.

Manager:

Well done, that's the spirit!

.-.-.-.-.-.-.-.-.-.-

The above scenario is a work of fiction and is not intended to represent any real events or persons.

Reminds me of the Iraqi Information Minister.


"Nothing is wrong here, nothing is wrong, everything is as it should be"

Andy
Can you comment on this EBU HD reference document, as it contradicts what you are saying:
Production Technology Seminar 2009 organised with the EBU Production Management Committee (PMC) EBU Headquarters, Geneva 27 - 29 January 2009

https://tech.ebu.ch/docs/events/production09/ebu-production09-report.pdf
Quote:
"Some experts felt strongly that even with the best encoder that 8 Mbit/s (1920x1080i / 25) is insufficient for HD broadcast of critical material. All experts felt strongly that with the best encoder, 6Mbit/s (1280x720p / 50) is insufficient for HDTV broadcast. Recommended minimum bit-rates “for critical material *but not unduly* so”:

 12,1 Mbit/s *MINIMUM* CBR for 1440 x 1080i/25
 12,8 Mbit/s minimum CBR for 1920 x 1080i/25(MPEG-2 24 Mbit/s reference)

Quality of the encoders has reached a mature level for various vendors and less drastic improvements in terms of picture quality are expected in the future. "
End Quote

One thing no one has commented on :

In Andy's diagram showing the comparison chain ... bottom right hand corner shows "Professional Decoder" .... err so what's that got to do with real life ?

We have a mixture of consumer units (in my case a 4 year old Sky HD box) ... so what's the PSNR of a consumer unit ?

Also the graphs show, if I'm interpreting then correctly, that poor source = worse PQ .... well that's exactly what we're complaining about ! Haven't you just proven our point?

I'm also concerned about the fact you are only looking at Luminance PSNR. That doesn't show the whole story I'm afraid, it is a valid test but should never be used in isolation. The URL provided by trevorjharris (https://www.compression.ru/video/quality_measure/info_en.html)is an excellent reference for comparison techniques between images and I suggest everyone has a good read. This is the sort of analysis I would have expect an organisation of the BBC's quality to have performed rather more than a very limited luminance PSNR comparison.

Andy

#41, 'and less drastic improvements in terms of picture quality are expected in the future', jeez I nearly wet myself

Andy,
I was expecting a list of programmes that the BBC claims looks better with the new encoder together with some screenshots.
Does such a list exist?
Will you please publish it? Or do we have to resort to the Freedom of Information Act again?

In the meantime let me give you a starter list of programmes that definately look worse at the reduced bitrate. In some cases I have recordings to prove it.
- Sunshine (drama)
- All natural history programmes (eg Natural World, South Pacific)
- Gavin and Stacey (personally I think Gavin and Stacey is case closed on this new encoder)

**All- please feel free to add to the above list - cut and paste and produce a longer list. Thanks

In fairness Andy's last 2 main paragraphs are a little more detailed than had been admitted so far:

"The new encoder produces images that correlate quite closely to the PSNR results. Programmes with low or no noise are noticeably better than they were on the old encoder. However, where the original images have noise we can see it on the new encoder's output but not on the old, suggesting that the new encoder is attempting to pass on more of the original image and confirming that a bit of noise reduction should be tested.

Dark pictures are inherently noisy, either because there is gain in the camera or the signal has been stretched too far in colour grading. We actually have a very noisy sequence that has too much camera gain and was stretched too far in post production. We used it during the tests to push the system, and even turning the bit rate up to just over 16Mbs made no difference to the image. We are trying a few new and different parameters that seem to improve noise handling and reduce the effect on screen. Again I will keep you posted."

---------------

But it still doesn't really admit to the blocking issues.

Trouble is key moments of high drama can often be in low light, as with Criminal Justice, Garrows Law etc. That degredation of key moments of tension can destroy an evenings viewing and emotionally increase the perception of the problem.

Had this slight admission of the problem been the starting point of some blog 4 months ago and had some real effort been made to improve the issues instead of the denial and name calling we've been subjected to, maybe just maybe, there would not be so much pent up frustration here - and now spilling outwards. We could have shared in a journey to improve things. Often a new version of software may come without some of the full feature set and that will be fixed with a new point release, we could have understood that, it's not unusual these days.

Someone somewhere should have realised when you've dug yourself into in a hole - stop digging.

The Independent is covering the story as well

https://www.independent.co.uk/news/media/tv-radio/bbc-criticised-over-hd-picture-quality-1837376.html

The comments on the mail's website really do make you fear for the sanity of some people.

Andy,
Can you confirm that you have selected settings for the new encoder that optimise it for low bitrates and you are pre-filtering out detail from the picture before encoding? (in an attempt to avoid compression artefacts).

It seems very evident to me that the HD picture detail has vanished and thats why (unless the picture is absolutely still) , the picture looks like just good SD, not HD.

Even Antiques Roadshow looks worse.

Looking at the graphs - have I got this right? The BBC have spent money on new encoders , but rather than use them to improve picture quality, you are trying to match the picture quality of the old encoders- which you've already told us were out of date.

So we have NEW encoders, trying to give us the OLD picture quality?

That's not adding value for the licence fee payer is it? Even if you get some operating cost savings, we don't see any improvement on screen. And people were already complaining about the old encoders when you cut their bitrate from 20 to 16mpbs. And now you are operating at 9mbps when the EBU says 12 minimum for easy material. And all your competitors (including BBC HD Europe) are using higher bitrates to get better picture quality - often 15. Eurosport, who use the same encoders as you are, run them at 18mbps. Do you think they have money to burn or do you think they have recognised that HD is all about the detail in the picture.

How disappointing that my critical post I made earlier was removed.

It wasn't rude, it called no one any names, but was mearly critical of this whole episode.

Not only does the BBC not want to answer questions about poor BBC HD quality directly - now it doesn't want anyone critising them for not doing so either.

Driving people to voice their opinion in other forums where the BBC has no authority doesn't help the BBC, in fact it only makes their arguement weaker.

No doubt this will be removed too...

Can someone technical comment on the methodology used here?

I thought that to select a bitrate you need to do this: (excuse the simplistic language here, I've forgotten the correct terms):

1. Run unencoded source material to establish a benchmark
2. Adjust the bitrate on the new encoder in steps (say 1 mpbs)
3. Measure the deterioration in picture quality at each bitrate level.
4. Select the bitrate that gives 1 or 2 levels of deterioration, no more than this

I can't see from Andy's blog where different bitrates where compared on the new encoder- or am I missing something?

Andy,

There don't seem to be any tests on the core topic of all the complaints - BITRATES. i.e. whether restoring higher bitrates will improve the picture quality.

You mention one test on high noise material where you did vary bitrate but no tests on normal material -say sports or natural history which we know is difficult to encode.

Have you tested what happens to the picture quality when you increase the bitrate or must I go down the Freedom of Information Act route again?

If you haven't done tests that would suggest you put cost before picture quality on the HD channel.

So, have you tested what happens when you increase the bitrate?

If these questions aren't answered I suggest we go down the Freedom of Information act route - that way they will have to be answered in 20 working days at least.

1. Why was the higher bit-rate not sustainable?

2. Why can't we have a short period back on the higher bit-rate as a simple test?

3. Why can't the satellite transmissions be statistically multiplexed to make better use of the available bandwidth on the transponder?

4. Why is BBC HD being transmitted in Scandinavia at a higher resolution and a higher bit-rate than that being transmitted to the licence paying public in the UK?

5. Will the BBC be increasing the bitrate in the near future? Yes?No

6. Will the BBC be changing the picture format to 1920x1020? Yes/No

7. Will the BBC be introducing a variable bitrate? Yes/No

I've thought of a new question to add to the list if you don't mind:

1. Why was the higher bit-rate not sustainable?

2. Why can't we have a short period back on the higher bit-rate as a simple test?

3. Why can't the satellite transmissions be statistically multiplexed to make better use of the available bandwidth on the transponder?

4. Why is BBC HD being transmitted in Scandinavia at a higher resolution and a higher bit-rate than that being transmitted to the licence paying public in the UK?

5. Will the BBC be increasing the bitrate in the near future? Yes?No

6. Will the BBC be changing the picture format to 1920x1020? Yes/No

7. Will the BBC be introducing a variable bitrate? Yes/No

NEW:
8. What happened to picture quality when you tested higher bitrates on the new encoder?

In preparation for the Freeview DSO I purchased a Bush Freesat HD box (going cheap at Argos). I worked on the premise that something would go wrong so I would need another TV source and unfortunately I was right.
After watching the HD offerings from ITV and the BBc I wondered why the picture quality on BBC HD wasn't as good as ITV so 'Google is your friend' and here I am.
I've spent a week reading all the blogs, forum posts, message boards and now news items covering this subject and I'm beginning to think I've fallen down the rabbit hole.
It only needs the queen to shout "Off with their heads" at all the people imagining the problem and I'll know that I'm not in Kansas anymore.
I've read Andy's technical blogs this week with interest, I can just about keep up with the science but at the end of the day the science isn't what it's about. It's about what people see on their screens which in my case isn't very much (HD wise).
Somewhere in these blogs I read that the new encoders were 2:1 better than the old ones so that would mean the BBC could run at 8Mb/s without loss of quality, perhaps that's what they tried and found it didn't work so they upped the bit rate until they found a rate that looked ok to them?
Unfortunately it doesn't look that good to us.
On a lighter note, my cat thinks it looks real enough.
My cat's HD Life

@mikerb
I think that's pretty much what they did. The 2:1 figure comes from the EBU I think but they also state that 12.1 Mbps minimum is required for non intensive 1440 broadcasts. Bear in mind that the old BBC HD level 16mbps was a BBC , not EBU derived setting. This in itself was substandard setting IMO, so the BBC have tried to half a substandard setting in the siwtch from mpeg 4 to mpeg 2. And the result - the negative reaction from viewers we have seen over the past 4 months?

Sorry , above post I got mpeg 2 and 4 the wrong way around.

In post #56 mikerb wrote:
On a lighter note, my cat thinks it looks real enough.
My cat's HD Life

Bless!

All these blogs have been excellent Andy but 3 months too late.

We have now had 4 months of problems, mix fade, noise, etc etc and we are still only on firmware 1?? Give us some permanernt solutions and more updates please.

Andy surely you are out of your all day meetings now. (Would your time not be better spent improving picture quality) Please answer the 7 -8 questions.

Here is mine question 9. Why is the BBC transmitting a identical copy to HD on the same transponder CH 6945 what is the point of that????

Andy,

Thanks for the very interesting insight. Still, you can do all the lab tests in the world but there's no substitute for real life.

Just to check that I've not been completely mistaken though, and the tests were right all along, I just went into the living room to watch the start of Gavin and Stacey.

Did you see it yourself? The opening scene with Stacey on a mobile on the bus, her face was a blurred mess; it was really reminiscent of that earlier screen grab from Defying Gravity (that was remarked on so memorably in fireyab9's mega post #794 on DN's PQ Blog - if you're still reading firyab9 I hope that Op went well).

Andy, just one chance left to explain all tomorrow. I wasn't going to do this again - I think you've got the message on the 1st 8 questions already - but there were some new ones mentioned earlier I'd like to add-in, see below:

1-9 , as above

10. Were tests carried out at many different bitrates to determine which would be used?

11. How did you decide on the new bitrate and why was it chosen?

@SkyCaddie #60, I think your suggested Q9 was answered previously elsewhere, I think by @derek500, something to do with it being the same stream as BBC HD but a different ID or something. I´m afraid my laptop with all my records on it is on the blink at the moment.

Gavin and Stacey was shocking. Someone PLEASE go to The Sun newspaper abvout this.

Gavin & Stacey 320 x 240 at 1 Mbps absolute disgrace. Please do not insult us by showing shocking quality like this. It was not even low resolution iPlayer quality.

Please put someting else on next week or switch the channel off for a hour.

Everytime I think you have reached a new low, it just gets worse.

Danielle & Andy how do you sleep at night you have no shame.


Last week Wigan refunded their fans after losing 9-1.

We want our money back !!!!!

Apologies if you've already seen this article, or if someone has posted it (I don't recall seeing it) but it reiterates some of our questions (I assume it's referring to the same event where Graham Plumb's comments caused a stir last week, albeit on a slightly lesser scale than Ms Scott's): https://crave.cnet.co.uk/televisions/0,39029474,49304459,00.htm

Could someone please confirm that this issue is only relevant to BBC HD on sat? I'm on Virgin cable, where I'm told the channel is 1920x1080 MPEG2 (not MPEG4) running at around 17-18mbit/sec, and I haven't noticed any particular drop in quality. How is Virgin's feed delivered? Is it a re-encode of the MPEG4 version, or is it a dedicated MPEG2 encode, which wouldn't have been affected by these recent encoder upgrades?

Can someone please explain the point of these blogs???? Im not really sure what they are there for other than to try and make us think that we are making the picture quality drop up.

Even the newspapers are printing articles yet BBC HD still wont admit theres an issue.

Someone at BBC HD *eyes look towards Danielle Nagler and Andy Quested* has to show some responsibility.

Andy Quested was once a man who was well respected, sadly I dont think thats the case anymore.

What I think every poster on here should do is copy the same qustions we keep asking until we get some honest answers.

#64 Interesting article Paul, which raises a lot of questions:

1. Why would they run the new channels up to 17? Isn't that an admission that 9.7 is far too low?

2. "all denied that picture quality has dropped in real terms". What does "real terms" mean? Could someone define that for me?

3. "The 40 per cent data-rate reduction shouldn't be visible to anyone." Of course not. Silly of us to have noticed it.....

Gavin and Stacey: (As usual, click twice for the full sized image)

https://img402.imageshack.us/i/gavinandstaceydisco1v.jpg/

What can you say? But I can understand why I frames like the above are preferred to what is shown for, at best, the other eleven twelfths of the time. Next frame:

https://img193.imageshack.us/i/gavinandstaceydisco2.jpg/

But hang on, this is a difficult scene, with flashing lights and fast movement. How about something less challenging, like blokes walking into a room in broad daylight?

https://img109.imageshack.us/i/gavinandstaceydoor.jpg/

I frames aren't really saving the day here either, are they?

Despite all this, I would rather The Sun didn't get involved. This is without doubt a hole the BBC have dug for themselves, but as one of their supporters I'd rather they respond to all these complaints by starting to climb out, rather than having us kick the dirt in on top of them. IMO.

Just to add, I didn't watch G&S as it's not my thing. I did pick these examples after a couple of minutes' whizzing through it in the editor, but they are not special. The programme is more or less composed of artefacted scenes like these, with breathers where there is little picture change. Of the three images I think number 2 is the most shocking, since it shows what the 'difficult' scenes look like for most of the time, and these surely have to be codec artefacts rather than anything further up the chain--don't they?

At least the video game the lads were playing looked OK.

It is a shame the witch-hunt some of you are embarking on towards Andy, Danielle, and the BBC in general has led to an absence of Andy replying to comments in the blog.

Not one person is saying they prefer the new encoders at the lower bitrate. I'd guess everyone at the BBC would have got this message by now. Harping on about it constantly and trying to bully them into some admission isn't particularly helpful.

If the BBC were to provide the best quality possible, they'd broadcast at a really high bitrate like 30mb/s. Quite obviously commercial aspects play a part in this and some trade-off in quality versus cost has to be achieved.

Andy is clearly a technical guy, and I'd guess not the kind of person making those balancing decisions. Hence why he hasn't commented on some of the questions some of you have asked.

I guess my point is it would be a shame to scare Andy away and have him refrain from commenting in this blog. A bit of politeness wouldn't go amiss. He's been pretty frank and honest from what I can see.

Not all of the questions have been answered, but some certainly have, and I find it refreshing for such technical details to be shared with us. If the shoe was on the other foot and the issue was with Sky One, I can't see Sky providing the information Andy has done.

Were people really expecting this blog to conclude tomorrow by saying "So yes we agree the quality is not good enough, and from tomorrow we will be increasing the bitrate..."?!

I think what this blog does show is that Andy has been doing his best to achieve a higher quality in both sound, vision, and production across the HD processing chain, and for that surely he must be congratulated. Hopefully he will continue this process, and iron out the problems with the new encoded that have resulted in a perceived drop in quality.

Well earlier today - abiout 7PM - I saw some very good closeup images of a face during one show one show.

It was the young girls face in closeup in Doctor Who 'Silence in the Library'

Hang on, THAT was on BBC THREE!

Cherrs, daveac - hoping tomorrow will have something to offer us in the hope of improvement on the BBC HD output.

Just looking at the third picture in post #68 and it seems to be an interlace problem, a slow shutter speed on the camera or both.

#72 Totally agree with the Sky comment.

Andy has been really helpful and I think people should tone down their comments somewhat.

#74 Funnily enough, I just saw a bit of Gavin and Stacey on BBC 3. Aside from the lurid DOG (who could watch this?), the typical problems in frames 1 and 3 above look to be present. Only frame 2 looks like an HD encoder issue; the rest is with the source, and urgent attention is needed to the interactions between the elements of the production and transmission chain. I'm guessing the same encoder is being used for SD, but whatever, it needs to be fixed.

G&S is not alone. On HD, The History of Christianity had the same allergy to motion; there was just less of it.

@HD_fan428 I too thought your pictures, and the actual programme, were quite shocking and not what I would expect from something purporting to be High Definition from our national broadcaster. Especially after all those tests we've learned about today, which proved conclusively that "the new encoder is doing better than the old" - or did they?

Surely it's not just my preconception about bit-rate changes that's affecting my opinion, i.e. that those pictures (and that programme) do not deserve the label HD (so clearly plastered all over them in the form of the DOG).

Really, now, I'm just hoping that tomorrow Andy's Blog is going to answer all those questions honestly, in full, and then go on to explain how the BBC now intend to "climb out" of the hole and to start delivering on their promises of "exceptionally clear, crisp pictures with vivid colours and up to five times more detail than standard definition" on the "Freesat Viewers' choice Channel of the Year" BBC HD.

Andy, I trust you to deliver on your promises and I'm sure that you realise that if you don't do that tomorrow then there are going to be an awful lot of disappointed Blog readers and HD Viewers out here. Perhaps a few outbursts in the comments will come your way too, so I only hope that you are permitted to put forward a solution for the channel, and not under instructions to tow a particular line or just obfuscate the issue with more smoke and mirrors (or even daggers! @Jeremy). Let's wait and see.

72, @GarySargent just read your post and agree with some of what you've said. It may seem there's a witch hunt from us but, honestly, I've got nothing against the HD team. My objection is that the channel has previously proven it can deliver outstanding quality, and now it doesn't any longer. Instead of explaining why it can no longer do so or, otherwise, why we've had a temporary hiccup but how it intends to return to its former capability, and in what time-scale, we seem to have been subjected to a lengthy and drawn-out denial that anything has changed.

Danielle and Andy are, quite rightly, the public face of the channel and so, whether or not they've made the decisions unfortunately they are going to get the flag. Your point is a good one though about not scaring them off, I certainly don't want to do that and do very much appreciate them (particularly Andy) actually engaging with us regularly (as I've, hopefully, told them quite often).

As for $ky, I've never been a big fan and I'm sure they don't engage with customers so readily - the point is though, I haven't heard a bad word said about the $ky HD experience. The same can't be said, these days, about the BBC HD experience.

for flag read flak - sorry

Have watched these blog comments and views for some time now with interest and, I'm afraid, dismay! Many thanks to Andy for the time and effort he is/has put in regarding his current technical lowdown etc.

I do not currently own HD equipment but have followed tech developments very closely in this area for many years.

I must, however, put in my two penneth worth with respect to a short visit I had only a couple of days ago to a local dedicated Panasonic dealer. It had multiple latest freesat capable plasma displays on show ( decently set up I think too ) that all appeared to be showing the same feed of BBC HD preview material. It was a recent BBC proms Robbie Williams set. I cannot exclude the possibility that they were all receiving the same feed from one of the latest freesat bluray/Harddrive recorders and that it was a recording of said programme in a lower bit rate mode but I do not think this was the case. Please, please, please someone tell me that what I was seeing ( on all the panels equally ) is not representative of recent BBC HD picture quality. Most relatively still shots seemed pretty high quality / sharp but when ANY, repeat, ANY movement started, even just the movement of Williams face side to side in close-up shots for instance, for the duration of that movement the picture turned into BLURO-VISION i.e. all detail was lost and only came back when virtually no movement was again the main type of signal!! ( And these were high quality plasmas not old crappy lcds! ). I could not believe that this could be an acceptable type of HD picture. I do not wish to exagerate or to offend anyone but you would have to be almost blind not to see this very, very bad motion problem. It wasn't really pixellation just an instantaneous blurring during any period of motion. I desperately hope someone is going to tell me that I had the unfortunate luck to be looking at BBC HD when it was showing something with a known bad picture issue associated with it. If not and this is going to be a representative type issue constantly then my hopes for HD broadcasting on Freesat and Freeview HD are vanishing down the toilet very quickly! I would rather have a highish bitrate SD MPEG 2 image than the mess I viewed the other day.

To Andy - I get the fundementals of your explanation regarding technical measurements of PSNR of images from the latest encoders but do these measurements have anything to say at all when it comes to the final received picture when motion becomes a critical part of the image quality??

I am seriously hoping that these current issues resolve themselves in an extremely positive way rather quickly, but, I must admit, I am not hopeful at the moment. If not, then I may give up on the idea of high quality public service broadcast TV and be forced to return to Transylvania and spend my days lying in a velvet lined coffin instead!

@72
>'Were people really expecting this blog to conclude tomorrow by saying "So yes we agree the quality is not good enough, and from tomorrow we will be increasing the bitrate..."?!'

After reading various comments, I'm not sure what people were expcting. I doubt they were really expecting that, although I think most people hoped that.

I do, hwoever, have the suspicion that that's not only what people want to hear, but te *only* thing people want to hear. I've thought this for several months of following these blogs, and found it quite interesting that Danielle got her head bitten off for drawing the same conclusion.

And after reading the comments on this particular, I'm perosnally curious about going over the comments to these and that post to see if any of the people demanding (here) nothing but a bitrate increase are any of the same people complainig (there) of being accused of that.

80, @Tiggs, thanks for that interesting comment. I wondered if I'm one of those people you refer to in your last paragraph? Talking of going over people's earlier comments, have you managed to get yourself an HD system yet or are you still working on the opinions you formed in Comet?

Now The Daily Telegraph has coverage of BBC HD.

Link to article here

Also @Tiggs

FWIW I went way out of my comfort zone and after pestering from their production team to film an insert for Points of View where I specifically referred to both bit rate and encoder changes causing a decrease in quality.

I discovered I was not alone in my experience but was, and continue to be, amazed by the tenor of the denial and siege mentality forthcoming from those in charge of the channel and some of its distribution methods.

I could not care less whether my appreciation index for key programming is returned to what it had been by increased bitrates new encoders or whatever.

I believe that the BBC should be the gold standard for quality and it genuinely distresses me to see BBCHD being talked about in such a terms as it is now but I don't blame the complainers.

@81

Actually, Paul, you're not one of those people I'm referring to. You're actually trying to get things addressed, rather than merely complaining on a forum/blog.

As for getting HD, I'm stuck until I next move it seems.
By the time I knew I didn't have to move this summer, it was less than Virgin's minimum-contract-with-early-termination-fee. And I cannot get a dish put up in my current flat.

82, @Mike, I saw my name in the Telegraph article you referred to, although I don't remember ever saying specifically what Urmee quoted me as saying. I think she's paraphrased something from @orion30 and attributed it to me. An understandable mistake since I have previously drawn her attention to how angry people (regular punters as described by Ms Scott of Freesat) have been getting on the Blogs, citing that post (DN PQ 502) as an example.

84 @Tiggs, thanks for your understanding. I wish you every success in getting HD in some shape or form in the near future.

Dear Andy,

Thank you for coming back to us - and I really look forward to tomorrow's Blog (I hadn't realised you intended to do another when I made my last post on Danielle's latest).

Paul

SkyCaddie @ 60 and paul_geaton @61 (last paragraph)

I have just checked the Astra 2D 10847 MHz transponder and there is in reality only one BBC HD vision stream. There are two HD 'Services'. The BBC HD 'Service PID 6940' links to 'Video PID 5500', 'Audio PID 5501' (ac3) and an empty 'Teletext PID 5503'. The other, 'Service 6945', links to exactly the same Video PID but takes as its default 'Audio PID 5503' - carrying stereo with Audio Description).

On that transponder there is also BBC1 East(W) operating at 4.3Mbps fixed, BBC1 Channel Islands operating at the same bit rate and the BBC HD channel at 9.7 Mbps fixed. All the audio channels (four in total) account for another 1.1 Mbps. Excluding a modicum of 'housekeeping' data that totals 19.4 Mbps. The transponder bandwidths are 33 Mhz but more Mbps can be fitted into the channel than the corresponding number of Mhz due to the coding system.

The BBC are leasing this transponder (along with 6 others) from SES-Astra and yet it is only half utilised. Nothing has changed from the time when this same transponder carried the same two BBC1 SD channels and the original 19 Mbps BBC HD.

There are further gains in transponder bit carrying capacity by changing over to DVB-S2 MPEG-4 as Sky already do. Unfortunately it isn't possible to carry DVB-S MPEG-2 and DVB-S2 MPEG-4 on a transponder at the same time. The standard for SD Freesat is DVB-S so that cannot be easily changed. The only other alternative is to move the two SD channels on to other transponders thereby reducing their bit rates even further. But this is not necessary. I think there is enough capacity for two 16 Mbps MPEG 2 HD channels on this transponder as well as the SD channels.

I should be very surprised if the new encoders, which have caused all the grief, cannot be set to encode the video at 16 or even 19 Mbps. I would suggest there is no obstacle, other than political, to operating the new encoders at higher bit rates even if this is only for a trial period.

I haven't even caught up yet with the final blog from Andy. I hope it is worth waiting for.

90, @rmgalley thanks so much for confirming all that. Pretty much what I thought but great to have it all explained so eloquently. I hope Andy comes back to this Blog tomorrow, and catches your comment - I'd love him to remark on what you've said, if only he could.

Don't hold your breath for anything enlightening from Andy's Friday Blog (nor Danielle's if you haven't caught up with that yet either) - it's the honest answers from Andy to the resulting comments that I'm waiting out for now.

Someone, somewhere, reading these Blogs must have inside knowledge of the mysterious "elephant in the room". I wish they'd spill the beans.

PSNR no longer applies in a modern h264 codec.

#92, @dansus - As a developer for the popular open source h264 codec, could you elaborate for me as a layman, in simple terms what that statement means.

When I take the BBC's latest evidence from their tests, which they tell us prove that picture quality hasn't suffered, to the BBC Trust, I would very much like to be able to point out if there is a fundamental flaw with those tests.

Are you telling me that these tests aren't conclusive? If so, would you provide me with more details please. Contact me here: https://www.zen97962.zen.co.uk/

This comment was removed because the moderators found it broke the house rules. Explain.

All this user's posts have been removed.Why?