Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

Content negotiation fails for some */* headers (regression)

#6022

On Rails 3.0.3 clients that have an accept header with */* in it will not get a response iff there is an unrecognized mime type in the header and there is no html/text mime type. For example (headers taken from actual production log)

These clients see a "template missing":

  • HTTP_ACCEPT: text/*
  • HTTP_ACCEPT: */*, auth/sicily
  • HTTP_ACCEPT: */*, application/youtube-client

These clients get the proper response:

  • HTTP_ACCEPT: */*
  • HTTP_ACCEPT: */*, text/html, application/youtube-client

These odd headers are pretty rare (Windows CE phone, down jones spider, etc), about 3 per day on a high traffic site, but according to specs they should still receive an HTML response as this ticket discusses.

Reported by Fjan · November 20th, 2010 @ 08:37 PM

State: resolved
Milestone: none
Assigned to: Neeraj Singh Neeraj Singh
Importance: Low

Activity

  1. Fjan
    Fjan

    Here is a monkey patch that you can drop into your rails initializers if you are in a hurry to fix this because you have a client with a mobile phone who really needs access. I'm fully aware this is not a proper solution but it might help someone.

    
    module ActionDispatch
      module Http
        module MimeNegotiation
    
          # Patched to always accept at least HTML
          def accepts
            @env["action_dispatch.request.accepts"] ||= begin
              header = @env['HTTP_ACCEPT'].to_s.strip
    
              if header.empty?
                [content_mime_type]
              else
                Mime::Type.parse(header) << Mime::HTML
              end
            end
          end
    
        end
      end
    end
    

    November 20th, 2010 @ 08:43 PM

  2. José Valim
    José Valim
    • Assigned user set to José Valim
    • Importance changed from to Low

    November 20th, 2010 @ 11:55 PM

  3. Neeraj Singh
    Neeraj Singh
    • State changed from new to open

    This patch takes care of all the three cases you outlined. https://github.com/rails/rails/compare/c937ddb...21fd93c

    HTTP_ACCEPT: text/
    HTTP_ACCEPT:
    /, auth/sicily
    HTTP_ACCEPT:
    /*, application/youtube-client

    More work is needed. But that's a start. Please check if that patch fixes your problem.

    November 22nd, 2010 @ 10:42 PM

  4. Fjan
    Fjan

    Thank you for that, that does indeed seem to cover all the cases.

    I've never seen a browser ask for image/ or application/ though, so we could make it easier on ourselves and assume any client that asks for a page with an HTML view can accept an HTML view. Simplifies the code and might be a bit more robust in serving clients that have an accept header that is plain/wrong. ("Do we want to be right or do we want to be pragmatic?"). Since you've gone through the trouble of creating this patch already I say being right is fine :-)

    November 23rd, 2010 @ 10:34 AM

  5. Repository
  6. Neeraj Singh
    Neeraj Singh
    • State changed from resolved to open

    I guess I put in wrong ticket # in the previous fix. Opening this ticket.

    November 24th, 2010 @ 01:28 AM

  7. José Valim
    José Valim
    • Assigned user changed from José Valim to Neeraj Singh

    I believe this can be closed, right Neeraj?

    November 25th, 2010 @ 09:46 AM

  8. Neeraj Singh
    Neeraj Singh
    • State changed from open to resolved

    November 25th, 2010 @ 05:52 PM

  9. Neeraj Singh
    Neeraj Singh

    Just listing the list of changes done as part of second set of changes.

    https://github.com/rails/rails/compare/c6030e8...fa2a5ae

    November 26th, 2010 @ 09:49 PM

  10. Repository
  11. Kenn Ejima
    Kenn Ejima

    I have another case for the same error that still fails with this commit

    • HTTP_ACCEPT: */*;q=0.01

    There's a semicolon in the string, the user agent was Mozilla/4.0 (PSP (PlayStation Portable); 2.00).

    January 8th, 2011 @ 06:14 AM

  12. Kenn Ejima
    Kenn Ejima

    This one should fall back to html (or plain text), too. It was sent from Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1).

    • HTTP_ACCEPT: text/*

    January 8th, 2011 @ 08:29 AM

  13. Neeraj Singh
    Neeraj Singh

    can you describe what error msg you are getting and what's the setup?

    January 9th, 2011 @ 12:07 AM

  14. Kenn Ejima
    Kenn Ejima

    On Rails 3.0.3 + Ruby 1.9.2, the same "template missing" error as Fjan described.

    Now my patch has the following regex in ActionDispatch::Http::MimeNegotiation:

    BROWSER_LIKE_ACCEPTS = /,\s*\*\/\*|\*\/\*\s*,|\*\/\*;|text\/\*/

    however I'm not sure if it's a good idea to keep adding exceptions in this manner - probably we should just at least try to find html template rather than throwing an exception? I don't know if there's a good reason when "template missing" is useful on production.

    January 9th, 2011 @ 02:17 AM

  15. Fjan
    Fjan

    Yes, I actually ended up using the monkey patch above that just return the HTML template if no other is available. There are just to many odd headers out there.

    January 9th, 2011 @ 09:13 AM

  16. Joel Chippindale
    Joel Chippindale

    Keen Ejima,

    The bug you mentioned with

    HTTP_ACCEPT: */*;q=0.01
    

    Is covered by #5833 Bug in Mime::Type#parse for single media range with accept params which includes a patch to fix.

    J.

    February 7th, 2011 @ 12:59 PM

  17. William Denniss
    William Denniss

    I think the monkey patch above isn't a bad idea actually – as a fallback. Maybe an option like this could actually be considered for inclusion? Otherwise, aren't we forever having to patch rails for every obscure accepts header found in the wild?

    Does returning status code 500 even make sense for these "bad" (in ruby's eyes) requests?

    I'm pretty happy with the idea "clients that know what they are doing can request alternative formats – everyone else gets html".

    April 2nd, 2011 @ 03:11 AM