Bdtyktl

For an odd reason my OLED TV can play HEVC 10-Bit files but only H.264 non-10-Bit.

This output (video stream) from ffmpeg is played fine with HEVC 10-Bit:

Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default)

while this file won't play:

Stream #0:0: Video: h264 (High 10), yuv420p10le(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

and this encode seems to be the max that will work on my TV:

Stream #0:0(jpn): Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

I use the following command to convert to HEVC:

ffmpeg -i in.mkv -map 0:0 -map 0:2 -map 0:4 -vcodec hevc -preset ultrafast -x265-params lossless=1 -acodec copy -scodec copy out.mkv

That of course takes a while and require the lossless switch or video gets extremely pixelated. I tried to look at hevc_nvenc but it said “no compatible devices” (maybe need to provide more info for it, I have an NVIDIA GeForce 980 TI).

Also tried using -vcodec h264 with -preset high and got an error saying:

Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

Any encoding gurus able to help with faster conversion to HEVC or conversion to a lower preset of H.264, or help getting NVENC to work if it provides significant performance gains…?

TVs are sometimes very picky about what profiles and levels they support for a given codec.

In any case, you should not have to resort to lossless encoding to retain quality when converting from H.264 to H.265. Please read the H.265 encoding guide for the available options w.r.t. quality and conversion speed.

While setting the ultrafast preset will definitely improve the overall speed of the encode, it'll significantly increase the file size (when using a constant quality encoding mode) or decrease the video quality (when using a target bitrate).

For HEVC, this should work:

ffmpeg -i <input> -c:v libx265 -crf 28 -preset fast <output>

Decrease the CRF value to increase the quality. A value of ±6 should result in half or double the bitrate, but it'll depend on the complexity of the content you're encoding.

I tried to look at hevc_nvenc but it said no compatible devices (maybe need to provide more info to it, I have 980Ti)

The 980 TI supports NVENC (see complete list).

Generally though, once you have a compatible device, in order to get support for hevc_nvenc, the NVIDIA libraries have to be installed on your system, and your ffmpeg needs to be compiled with support for nvenc. See this wiki entry for more info.

conversion to a lower preset of h264

This may help, but I assume that you want to keep the 10-bit color depth in your files? If you're converting, you might as well convert to HEVC – I wouldn't recommend converting to 8-bit H.264.