JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

kerickson
I'm using TomEE webprofile 7.04 in an exploded WAR deployment. I need to use JsonReader's readObject() to read from a JsonReader created from a string, but the JsonReader's readObject() call fails ineloquently when the object read is too long. I unfortunately haven't been able to find a way to make a unit test that reproduces this, it always passes even if the string length is over this limit.. I think being scoped to a request may have something to do with it.

After putting the attached bad.json in my user directory, I acquire the attached file's contents as such:

File bad = new File(System.getProperty("user.dir")+"/bad.json");
byte[] badbytes = Files.readAllBytes(bad.toPath());
String badStr =  new String(badbytes, StandardCharsets.UTF_8);
try(JsonReader jr = Json.createReader(new StringReader(badStr))) {
    JsonObject json = jr.readObject();
}

From this I get the error:

"java.lang.ArrayIndexOutOfBoundsException: 8192
    at org.apache.johnzon.core.JsonStreamParserImpl.appendToCopyBuffer(JsonStreamParserImpl.java:156)
    at org.apache.johnzon.core.JsonStreamParserImpl.readString(JsonStreamParserImpl.java:581)
    at org.apache.johnzon.core.JsonStreamParserImpl.handleQuote(JsonStreamParserImpl.java:684)
    at org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:429)
    at org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:389)
    at org.apache.johnzon.core.JsonReaderImpl.parseObject(JsonReaderImpl.java:181)
    at org.apache.johnzon.core.JsonReaderImpl.readValue(JsonReaderImpl.java:82)
    at org.apache.johnzon.core.JsonReaderImpl.read(JsonReaderImpl.java:61)
    at org.apache.johnzon.core.JsonReaderImpl.readObject(JsonReaderImpl.java:150)"


....but if I replace all the backslashes in that file with a standard character like "a", I get the error message that is probably intended for the above case:

Too many characters. Maximum string/number length of 8192 exceeded on [lineNumber=1, columnNumber=10591, streamOffset=10590]. Maybe increase org.apache.johnzon.max-string-length in jsonp factory properties or system properties.


I think there's a bug where this message doesn't appear if when there are a lot of escape characters (slashes) that bring the string length over the limit. Any reason this proper warning wouldn't appear when triggered by escape characters? This is occurring both on Windows and Unix.

Thanks,
-Kean

bad.json (22K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

kerickson
Sorry to reply to my own email but one more question on that--if I am to
set "org.apache.johnzon.max-string-length", is there anywhere that the
default value for this is documented? The error message claims the default
is 8192, but if I override it in system.properties with 4092 the warning
goes away even though that would have resulted in the limit being lowered
if the "Too many characters" error message were correct.. So I'm not really
sure what's going on.

On Mon, Feb 25, 2019 at 5:19 PM Kean Erickson <[hidden email]>
wrote:

> I'm using TomEE webprofile 7.04 in an exploded WAR deployment. I need to
> use JsonReader's readObject() to read from a JsonReader created from a
> string, but the JsonReader's readObject() call fails ineloquently when the
> object read is too long. I unfortunately haven't been able to find a way to
> make a unit test that reproduces this, it always passes even if the string
> length is over this limit.. I think being scoped to a request may have
> something to do with it.
>
> After putting the attached bad.json in my user directory, I acquire the
> attached file's contents as such:
>
> File bad = new File(System.getProperty("user.dir")+"/bad.json");
> byte[] badbytes = Files.readAllBytes(bad.toPath());
> String badStr =  new String(badbytes, StandardCharsets.UTF_8);
> try(JsonReader jr = Json.createReader(new StringReader(badStr))) {
>     JsonObject json = jr.readObject();
> }
>
> From this I get the error:
>
> "java.lang.ArrayIndexOutOfBoundsException: 8192
>     at
> org.apache.johnzon.core.JsonStreamParserImpl.appendToCopyBuffer(JsonStreamParserImpl.java:156)
>     at
> org.apache.johnzon.core.JsonStreamParserImpl.readString(JsonStreamParserImpl.java:581)
>     at
> org.apache.johnzon.core.JsonStreamParserImpl.handleQuote(JsonStreamParserImpl.java:684)
>     at
> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:429)
>     at
> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:389)
>     at
> org.apache.johnzon.core.JsonReaderImpl.parseObject(JsonReaderImpl.java:181)
>     at
> org.apache.johnzon.core.JsonReaderImpl.readValue(JsonReaderImpl.java:82)
>     at org.apache.johnzon.core.JsonReaderImpl.read(JsonReaderImpl.java:61)
>     at
> org.apache.johnzon.core.JsonReaderImpl.readObject(JsonReaderImpl.java:150)"
>
>
> ....but if I replace all the backslashes in that file with a standard
> character like "a", I get the error message that is probably intended for
> the above case:
>
> Too many characters. Maximum string/number length of 8192 exceeded on
> [lineNumber=1, columnNumber=10591, streamOffset=10590]. Maybe increase
> org.apache.johnzon.max-string-length in jsonp factory properties or system
> properties.
>
>
> I think there's a bug where this message doesn't appear if when there are
> a lot of escape characters (slashes) that bring the string length over the
> limit. Any reason this proper warning wouldn't appear when triggered by
> escape characters? This is occurring both on Windows and Unix.
>
> Thanks,
> -Kean
>
Reply | Threaded
Open this post in threaded view
|

Re: JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

Mark Struberg-2
We did tons of fixes in Johnzon. Can you please try out TomEE-7.0.5? This version contains a much newer version of Johnzon.

LieGrue,
strub


> Am 26.02.2019 um 20:54 schrieb Kean Erickson <[hidden email]>:
>
> Sorry to reply to my own email but one more question on that--if I am to
> set "org.apache.johnzon.max-string-length", is there anywhere that the
> default value for this is documented? The error message claims the default
> is 8192, but if I override it in system.properties with 4092 the warning
> goes away even though that would have resulted in the limit being lowered
> if the "Too many characters" error message were correct.. So I'm not really
> sure what's going on.
>
> On Mon, Feb 25, 2019 at 5:19 PM Kean Erickson <[hidden email]>
> wrote:
>
>> I'm using TomEE webprofile 7.04 in an exploded WAR deployment. I need to
>> use JsonReader's readObject() to read from a JsonReader created from a
>> string, but the JsonReader's readObject() call fails ineloquently when the
>> object read is too long. I unfortunately haven't been able to find a way to
>> make a unit test that reproduces this, it always passes even if the string
>> length is over this limit.. I think being scoped to a request may have
>> something to do with it.
>>
>> After putting the attached bad.json in my user directory, I acquire the
>> attached file's contents as such:
>>
>> File bad = new File(System.getProperty("user.dir")+"/bad.json");
>> byte[] badbytes = Files.readAllBytes(bad.toPath());
>> String badStr =  new String(badbytes, StandardCharsets.UTF_8);
>> try(JsonReader jr = Json.createReader(new StringReader(badStr))) {
>>    JsonObject json = jr.readObject();
>> }
>>
>> From this I get the error:
>>
>> "java.lang.ArrayIndexOutOfBoundsException: 8192
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.appendToCopyBuffer(JsonStreamParserImpl.java:156)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.readString(JsonStreamParserImpl.java:581)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.handleQuote(JsonStreamParserImpl.java:684)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:429)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:389)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.parseObject(JsonReaderImpl.java:181)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.readValue(JsonReaderImpl.java:82)
>>    at org.apache.johnzon.core.JsonReaderImpl.read(JsonReaderImpl.java:61)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.readObject(JsonReaderImpl.java:150)"
>>
>>
>> ....but if I replace all the backslashes in that file with a standard
>> character like "a", I get the error message that is probably intended for
>> the above case:
>>
>> Too many characters. Maximum string/number length of 8192 exceeded on
>> [lineNumber=1, columnNumber=10591, streamOffset=10590]. Maybe increase
>> org.apache.johnzon.max-string-length in jsonp factory properties or system
>> properties.
>>
>>
>> I think there's a bug where this message doesn't appear if when there are
>> a lot of escape characters (slashes) that bring the string length over the
>> limit. Any reason this proper warning wouldn't appear when triggered by
>> escape characters? This is occurring both on Windows and Unix.
>>
>> Thanks,
>> -Kean
>>

Reply | Threaded
Open this post in threaded view
|

Re: JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

kerickson
I swapped to 7.1.0 and am seeing the same thing. But I think I have a
better idea of what is going on by default when this property isn't
configured manually.

In JsonParserFactoryImpl, a system level maxSize value is set and to
default to 262144 if the max-string-length johnzon property isn't set as a
system property. But before this, CxfRSService sets this in a system
property as 8192. So the larger value 262144 never applies.

On Tue, Feb 26, 2019 at 1:44 PM Mark Struberg <[hidden email]>
wrote:

> We did tons of fixes in Johnzon. Can you please try out TomEE-7.0.5? This
> version contains a much newer version of Johnzon.
>
> LieGrue,
> strub
>
>
> > Am 26.02.2019 um 20:54 schrieb Kean Erickson <[hidden email]>:
> >
> > Sorry to reply to my own email but one more question on that--if I am to
> > set "org.apache.johnzon.max-string-length", is there anywhere that the
> > default value for this is documented? The error message claims the
> default
> > is 8192, but if I override it in system.properties with 4092 the warning
> > goes away even though that would have resulted in the limit being lowered
> > if the "Too many characters" error message were correct.. So I'm not
> really
> > sure what's going on.
> >
> > On Mon, Feb 25, 2019 at 5:19 PM Kean Erickson <[hidden email]>
> > wrote:
> >
> >> I'm using TomEE webprofile 7.04 in an exploded WAR deployment. I need to
> >> use JsonReader's readObject() to read from a JsonReader created from a
> >> string, but the JsonReader's readObject() call fails ineloquently when
> the
> >> object read is too long. I unfortunately haven't been able to find a
> way to
> >> make a unit test that reproduces this, it always passes even if the
> string
> >> length is over this limit.. I think being scoped to a request may have
> >> something to do with it.
> >>
> >> After putting the attached bad.json in my user directory, I acquire the
> >> attached file's contents as such:
> >>
> >> File bad = new File(System.getProperty("user.dir")+"/bad.json");
> >> byte[] badbytes = Files.readAllBytes(bad.toPath());
> >> String badStr =  new String(badbytes, StandardCharsets.UTF_8);
> >> try(JsonReader jr = Json.createReader(new StringReader(badStr))) {
> >>    JsonObject json = jr.readObject();
> >> }
> >>
> >> From this I get the error:
> >>
> >> "java.lang.ArrayIndexOutOfBoundsException: 8192
> >>    at
> >>
> org.apache.johnzon.core.JsonStreamParserImpl.appendToCopyBuffer(JsonStreamParserImpl.java:156)
> >>    at
> >>
> org.apache.johnzon.core.JsonStreamParserImpl.readString(JsonStreamParserImpl.java:581)
> >>    at
> >>
> org.apache.johnzon.core.JsonStreamParserImpl.handleQuote(JsonStreamParserImpl.java:684)
> >>    at
> >>
> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:429)
> >>    at
> >>
> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:389)
> >>    at
> >>
> org.apache.johnzon.core.JsonReaderImpl.parseObject(JsonReaderImpl.java:181)
> >>    at
> >> org.apache.johnzon.core.JsonReaderImpl.readValue(JsonReaderImpl.java:82)
> >>    at
> org.apache.johnzon.core.JsonReaderImpl.read(JsonReaderImpl.java:61)
> >>    at
> >>
> org.apache.johnzon.core.JsonReaderImpl.readObject(JsonReaderImpl.java:150)"
> >>
> >>
> >> ....but if I replace all the backslashes in that file with a standard
> >> character like "a", I get the error message that is probably intended
> for
> >> the above case:
> >>
> >> Too many characters. Maximum string/number length of 8192 exceeded on
> >> [lineNumber=1, columnNumber=10591, streamOffset=10590]. Maybe increase
> >> org.apache.johnzon.max-string-length in jsonp factory properties or
> system
> >> properties.
> >>
> >>
> >> I think there's a bug where this message doesn't appear if when there
> are
> >> a lot of escape characters (slashes) that bring the string length over
> the
> >> limit. Any reason this proper warning wouldn't appear when triggered by
> >> escape characters? This is occurring both on Windows and Unix.
> >>
> >> Thanks,
> >> -Kean
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: JsonReader.readObject() fails with ArrayIndexOutOfBoundsException when a string is too long, but is full of backslashes?

kerickson
Sorry to bump this but it looks like this is a much more general problem than I thought. Previously I was having an issue when reading a file with too many backslashes, and the fix was to acquire my JsonReader from a JsonReaderFactory created with a settings map that overrides JsonParserFactoryImpl.MAX_STRING_LENGTH with a correct value so that the file read wouldn't explode.

But now I'm noticing that it's possible for a simple POST or PUT to a resource with a String value of the attached contents blows up with the same "java.lang.ArrayIndexOutOfBoundsException: 8192" error message from before, due to having just the right number of backslahes from tabs, only it doesn't seem like I have any control in this case to override the maximum length that is being used. Like I mentioned previously, I think the bug is that in JsonParserFactoryImpl, a system level maxSize value is set to default to 262144 if the max-string-length johnzon property isn't set as a system property. But before this, CxfRSService sets this in a system property as 8192. So the larger value 262144 never applies.

This unconfigurable string length limitation is a problem for any API we create that will use Johnzon to receive a string, do y'all know of any kind of workaround (like a system-level setting)?

Thanks,
-Kean

On Tue, Feb 26, 2019 at 5:45 PM Kean Erickson <[hidden email]> wrote:
I swapped to 7.1.0 and am seeing the same thing. But I think I have a better idea of what is going on by default when this property isn't configured manually.

In JsonParserFactoryImpl, a system level maxSize value is set and to default to 262144 if the max-string-length johnzon property isn't set as a system property. But before this, CxfRSService sets this in a system property as 8192. So the larger value 262144 never applies.

On Tue, Feb 26, 2019 at 1:44 PM Mark Struberg <[hidden email]> wrote:
We did tons of fixes in Johnzon. Can you please try out TomEE-7.0.5? This version contains a much newer version of Johnzon.

LieGrue,
strub


> Am 26.02.2019 um 20:54 schrieb Kean Erickson <[hidden email]>:
>
> Sorry to reply to my own email but one more question on that--if I am to
> set "org.apache.johnzon.max-string-length", is there anywhere that the
> default value for this is documented? The error message claims the default
> is 8192, but if I override it in system.properties with 4092 the warning
> goes away even though that would have resulted in the limit being lowered
> if the "Too many characters" error message were correct.. So I'm not really
> sure what's going on.
>
> On Mon, Feb 25, 2019 at 5:19 PM Kean Erickson <[hidden email]>
> wrote:
>
>> I'm using TomEE webprofile 7.04 in an exploded WAR deployment. I need to
>> use JsonReader's readObject() to read from a JsonReader created from a
>> string, but the JsonReader's readObject() call fails ineloquently when the
>> object read is too long. I unfortunately haven't been able to find a way to
>> make a unit test that reproduces this, it always passes even if the string
>> length is over this limit.. I think being scoped to a request may have
>> something to do with it.
>>
>> After putting the attached bad.json in my user directory, I acquire the
>> attached file's contents as such:
>>
>> File bad = new File(System.getProperty("user.dir")+"/bad.json");
>> byte[] badbytes = Files.readAllBytes(bad.toPath());
>> String badStr =  new String(badbytes, StandardCharsets.UTF_8);
>> try(JsonReader jr = Json.createReader(new StringReader(badStr))) {
>>    JsonObject json = jr.readObject();
>> }
>>
>> From this I get the error:
>>
>> "java.lang.ArrayIndexOutOfBoundsException: 8192
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.appendToCopyBuffer(JsonStreamParserImpl.java:156)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.readString(JsonStreamParserImpl.java:581)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.handleQuote(JsonStreamParserImpl.java:684)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:429)
>>    at
>> org.apache.johnzon.core.JsonStreamParserImpl.next(JsonStreamParserImpl.java:389)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.parseObject(JsonReaderImpl.java:181)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.readValue(JsonReaderImpl.java:82)
>>    at org.apache.johnzon.core.JsonReaderImpl.read(JsonReaderImpl.java:61)
>>    at
>> org.apache.johnzon.core.JsonReaderImpl.readObject(JsonReaderImpl.java:150)"
>>
>>
>> ....but if I replace all the backslashes in that file with a standard
>> character like "a", I get the error message that is probably intended for
>> the above case:
>>
>> Too many characters. Maximum string/number length of 8192 exceeded on
>> [lineNumber=1, columnNumber=10591, streamOffset=10590]. Maybe increase
>> org.apache.johnzon.max-string-length in jsonp factory properties or system
>> properties.
>>
>>
>> I think there's a bug where this message doesn't appear if when there are
>> a lot of escape characters (slashes) that bring the string length over the
>> limit. Any reason this proper warning wouldn't appear when triggered by
>> escape characters? This is occurring both on Windows and Unix.
>>
>> Thanks,
>> -Kean
>>


sample-html.txt (12K) Download Attachment