Change-Id: Ic7701a9e4635fe1a331c9a1df776ed580759eb9d
Reviewed-on: https://go-review.googlesource.com/c/go/+/268758
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Trust: Nigel Tao <nigeltao@golang.org>
Trust: Alberto Donizetti <alb.donizetti@gmail.com>
Trust: Rob Pike <r@golang.org>
As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)
Update the Go tree to use the preferred names.
As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.
ReadDir changes are in a separate CL, because they are not a
simple search and replace.
For #42026.
Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The old ioutil references are still valid, but update our code
to reflect best practices and get used to the new locations.
Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.
Also excluded vendored code.
For #41190.
Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1
Reviewed-on: https://go-review.googlesource.com/c/go/+/263142
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Each URL was manually verified to ensure it did not serve up incorrect
content.
Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Also add some b.ReportAllocs calls to other image codec benchmarks.
Change-Id: I0f055dc76bffb66329c621a5f1ccd239f0cdd30b
Reviewed-on: https://go-review.googlesource.com/68390
Reviewed-by: Jed Denlea <jed@fastly.com>
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Previously, the package did not distinguish between baseline and
extended sequential images. Both are non-progressive images, but the Th
range differs between the two, as per Annex B of
https://www.w3.org/Graphics/JPEG/itu-t81.pdf
Extended sequential images are often emitted by the Guetzli encoder.
Fixes#19913
Change-Id: I3d0f9e16d5d374ee1c65e3a8fb87519de61cff94
Reviewed-on: https://go-review.googlesource.com/41831
Reviewed-by: David Symonds <dsymonds@golang.org>
The existing implementation falls back to using image.At()
for each pixel when encoding an *image.YCbCr which is
inefficient and causes many memory allocations.
This change makes the jpeg encoder directly read Y, Cb, and Cr
pixel values.
benchmark old ns/op new ns/op delta
BenchmarkEncodeYCbCr-4 43990846 24201148 -44.99%
benchmark old MB/s new MB/s speedup
BenchmarkEncodeYCbCr-4 20.95 38.08 1.82x
Fixes#18487
Change-Id: Iaf2ebc646997e3e1fffa5335f1b0d642e15bd453
Reviewed-on: https://go-review.googlesource.com/34773
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Fixes#14522.
As I said on that issue:
----
This is a progressive JPEG image. There are two dimensions of
progressivity: spectral selection (variables zs and ze in scan.go,
ranging in [0, 63]) and successive approximation (variables ah and al in
scan.go, ranging in [0, 8), from LSB to MSB, although ah=0 implicitly
means ah=8).
For this particular image, there are three components, and the SOS
markers contain this progression:
zs, ze, ah, al: 0 0 0 0 components: 0, 1, 2
zs, ze, ah, al: 1 63 0 0 components: 1
zs, ze, ah, al: 1 63 0 0 components: 2
zs, ze, ah, al: 1 63 0 2 components: 0
zs, ze, ah, al: 1 10 2 1 components: 0
zs, ze, ah, al: 11 63 2 1 components: 0
zs, ze, ah, al: 1 10 1 0 components: 0
The combination of all of these is complete (i.e. spectra 0 to 63 and
bits 8 exclusive to 0) for components 1 and 2, but it is incomplete for
component 0 (the luma component). In particular, there is no data for
component 0, spectra 11 to 63 and bits 1 exclusive to 0.
The image/jpeg code, as of Go 1.6, waits until both dimensions are
complete before performing the de-quantization, IDCT and copy to an
*image.YCbCr. This is the "if zigEnd != blockSize-1 || al != 0 { ...
continue }" code and associated commentary in scan.go.
Almost all progressive JPEG images end up complete in both dimensions
for all components, but this particular image is incomplete for
component 0, so the Go code never writes anything to the Y values of the
resultant *image.YCbCr, which is why the broken output is so dark (but
still looks recognizable in terms of red and blue hues).
My reading of the ITU T.81 JPEG specification (Annex G) doesn't
explicitly say that this is a valid image, but it also doesn't rule it
out.
In any case, the fix is, for progressive JPEG images, to always
reconstruct the decoded blocks (by performing the de-quantization, IDCT
and copy to an *image.YCbCr), regardless of whether or not they end up
complete. Note that, in Go, the jpeg.Decode function does not return
until the entire image is decoded, so we still only want to reconstruct
each block once, not once per SOS (Start Of Scan) marker.
----
A test image was also added, based on video-001.progressive.jpeg. When
decoding that image, inserting a
println("nComp, zs, ze, ah, al:", nComp, zigStart, zigEnd, ah, al)
into decoder.processSOS in scan.go prints:
nComp, zs, ze, ah, al: 3 0 0 0 1
nComp, zs, ze, ah, al: 1 1 5 0 2
nComp, zs, ze, ah, al: 1 1 63 0 1
nComp, zs, ze, ah, al: 1 1 63 0 1
nComp, zs, ze, ah, al: 1 6 63 0 2
nComp, zs, ze, ah, al: 1 1 63 2 1
nComp, zs, ze, ah, al: 3 0 0 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0
In other words, video-001.progressive.jpeg contains 10 different scans.
This little program below drops half of them (remembering to keep the
"\xff\xd9" End of Image marker):
----
package main
import (
"bytes"
"io/ioutil"
"log"
)
func main() {
sos := []byte{0xff, 0xda}
eoi := []byte{0xff, 0xd9}
src, err := ioutil.ReadFile("video-001.progressive.jpeg")
if err != nil {
log.Fatal(err)
}
b := bytes.Split(src, sos)
println(len(b)) // Prints 11.
dst := bytes.Join(b[:5], sos)
dst = append(dst, eoi...)
if err := ioutil.WriteFile("video-001.progressive.truncated.jpeg", dst, 0666); err != nil {
log.Fatal(err)
}
}
----
The video-001.progressive.truncated.jpeg was converted to png via
libjpeg and ImageMagick:
djpeg -nosmooth video-001.progressive.truncated.jpeg > tmp.tga
convert tmp.tga video-001.progressive.truncated.png
rm tmp.tga
Change-Id: I72b20cd4fb6746d36d8d4d587f891fb3bc641f84
Reviewed-on: https://go-review.googlesource.com/21062
Reviewed-by: Rob Pike <r@golang.org>
This rolls back most of golang.org/cl/8841, aka 2f98bac310, and makes a
different fix. It keeps the TestTruncatedSOSDataDoesntPanic test
introduced by that other CL, which obviously still passes after this CL.
Fixes#11650, a regression (introduced by cl/8841) from Go 1.4.
The original cl/8841 changed the image/jpeg not to panic on an input
given in #10387. We still do not panic on that input, after this CL.
I have a corpus of over 160,000 JPEG images, a sample of a web crawl.
The image/jpeg code ran happily over that whole corpus both before and
after this CL, although that corpus clearly didn't catch the regression
in the first place.
This code was otherwise tested manually. I don't think that it's trivial
to synthesize a JPEG input that happens to run out of Huffman data at
just the right place. The test image attached to #11650 obviously has
that property, but I don't think we can simply add that test image to
the repository: it's 227KiB, and I don't know its copyright status.
I also looked back over the issue tracker for problematic JPEGs that
people have filed. The Go code, after this CL, is still happy on these
files in my directory:
issue2362a.jpeg
issue3916.jpeg
issue3976.jpeg
issue4084.jpeg
issue4259.jpeg
issue4291.jpeg
issue4337.jpeg
issue4500.jpeg
issue4705.jpeg
issue4975.jpeg
issue5112.jpeg
issue6767.jpeg
issue9888.jpeg
issue10133.jpeg
issue10357.jpeg
issue10447.jpeg
issue11648.jpeg
issue11650.jpeg
There were other images attached in the issue tracker that aren't
actually valid JPEGs. They failed both before and after this CL:
broken-issue2362b.jpeg
broken-issue6450.jpeg
broken-issue8693.jpeg
broken-issue10154.jpeg
broken-issue10387.jpeg
broken-issue10388.jpeg
broken-issue10389.jpeg
broken-issue10413.jpeg
In summary, this CL fixes#11650 and, after some automated and manual
testing, I don't think introduces new regressions.
Change-Id: I30b67036e9b087f3051d57dac7ea05fb4fa36f66
Reviewed-on: https://go-review.googlesource.com/12163
Reviewed-by: Rob Pike <r@golang.org>
call unreadByteStuffedByte.
If ensureNBits was due to an io.EOF that was translated to
jpeg.errShortHuffmanData, then we may have read no bytes, so there is no
byte-stuffed-byte to unread.
Fixes#10387
Change-Id: I39a3842590c6cef2aa48943288d52f603338b44d
Reviewed-on: https://go-review.googlesource.com/8841
Reviewed-by: Rob Pike <r@golang.org>
The imageutil.DrawYCbCr function lives in an internal package because it
is needed by both the image/draw and image/jpeg packages, but it doesn't
seem right for one of those two to depend on the other.
It could eventually go into the image package, but that would require
committing to an API for the rest of Go 1.x.
Change-Id: I7b12555c970d86409365e99eef9360702aaffa30
Reviewed-on: https://go-review.googlesource.com/7925
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
components.
This fixes decoding JPEG images where the component selector is 0. Such
images are rare, but not impossible.
Change-Id: I6d221bce01cce8cc0440e117543233371782ca22
Reviewed-on: https://go-review.googlesource.com/6421
Reviewed-by: Rob Pike <r@golang.org>
The new testdata was created by:
convert video-001.png -colorspace cmyk video-001.cmyk.jpeg
video-001.cmyk.jpeg was then converted back to video-001.cmyk.png via
the GIMP. ImageMagick (convert) wasn't used for this second conversion
because IM's default color profiles complicates things.
Fixes#4500.
Change-Id: Ibf533f6a6c7e76883acc493ce3a4289d7875df3f
Reviewed-on: https://go-review.googlesource.com/4801
Reviewed-by: Rob Pike <r@golang.org>