Problems with metadata-only requests and media MIME types

[AndrewMagliozzi]

Thanks for the recent updates to this npm package. It's great, but I've been having some issues with the GMail API since the update to version 1.0.0.

For at least the GMail API and possibly other apis as well, there is little support for metadata-only requests.

Looking at the api code, 'isMedia' is always set for true in a function like users.drafts.create, as seen here: https://github.com/google/google-api-nodejs-client/blob/master/apis/gmail/v1.js#L65. Also, the /uploads URL is always used although some APIs support metadata-only requests with a different URL.

A related issue is that it is impossible to correctly set the media MIME type for multipart uploads, which is all the code currently supports. The GMail users.drafts.create method, for example, requires a media MIME type of message/rfc822. If we make no effort to set this, we get the following API error:

Media type 'application/octet-stream' is not supported. 
Valid media types: [message/rfc822]

If, however, we add resource: { mimeType: 'message/rfc822' } to the options object, we get the following API error:

Unknown field name: mimeType

This is because the gmail API does not have a mimeType field. Unfortunately, this is where the createAPIRequest function looks for setting the media MIME type. This seems to work for some APIs, such as drive, which actually do have a mimeType field, but it does not work for gmail.

Thanks for any help you can provide in resolving this issue.

[ryanseys]

Hi @AndrewMagliozzi, thanks for the report. I'll look into this.

[rakyll]

We should set isMedia only if media is given or create multipart if media resource exists. It looks like a quick fix.

[AndrewMagliozzi]

Hi @rakyll that's a great start, but it's only part of the problem as I see it.

The URL would also need to be modified, because 'upload' is hard-coded in the api endpoint in places like this: https://github.com/google/google-api-nodejs-client/blob/master/apis/gmail/v1.js#L59

This is particularly relevant for api calls like this one where there are two end points (one for upload and one for metadata-only): https://developers.google.com/gmail/api/v1/reference/users/drafts/create

Thanks for your fast replies!

[ryanseys]

Yep. We will have to take into consideration both URLs. I'm working on a fix.

[ryanseys]

"/upload" is just appended to the end of every URL so we can just trim it off for the short term.

[rakyll]

It's prepended to the path. I'd not rely on hard-coded find and replace though. Discovery doc provides paths for media and non-media endpoints.

[AndrewMagliozzi]

There may still be an issue with adjusting the mimetype as well. createAPIRequest may need some adjusting too.

This is where the mimeType gets set: https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L121

Unfortunately, the GMail API returns this error when I try to specify a different mimetype in the options object:

Unknown field name: mimeType

You may want to address the mimeType param like you do here with some of the other parts of the options object: https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L78-L84

[atrigent]

I've been looking at the discovery API, and I don't see where it defines the distinction between metadata-only and upload. As far as I can tell, API methods can have either 1. only non-upload URLS, 2. both non-upload and upload URLs, or 3. only upload URLs. Here are examples of these three cases:

1. Only non-upload: https://developers.google.com/drive/v2/reference/files/copy. Here is the description I get for this method from the discovery API:

{
    "id": "drive.files.copy",
    "path": "files/{fileId}/copy",
    "httpMethod": "POST",
    "description": "Creates a copy of the specified file.",
    "parameters": {
        ...
    },
    "parameterOrder": [
        "fileId"
    ],
    "request": {
        "$ref": "File"
    },
    "response": {
        "$ref": "File"
    },
    "scopes": [
        "https://www.googleapis.com/auth/drive",
        "https://www.googleapis.com/auth/drive.appdata",
        "https://www.googleapis.com/auth/drive.apps.readonly",
        "https://www.googleapis.com/auth/drive.file"
    ]
}

1. Upload and non-upload: https://developers.google.com/drive/v2/reference/files/insert.

{
    "id": "drive.files.insert",
    "path": "files",
    "httpMethod": "POST",
    "description": "Insert a new file.",
    "parameters": {
        ...
    },
    "request": {
        "$ref": "File"
    },
    "response": {
        "$ref": "File"
    },
    "scopes": [
        "https://www.googleapis.com/auth/drive",
        "https://www.googleapis.com/auth/drive.appdata",
        "https://www.googleapis.com/auth/drive.apps.readonly",
        "https://www.googleapis.com/auth/drive.file"
    ],
    "supportsMediaUpload": true,
    "mediaUpload": {
        "accept": [
            "*/*"
        ],
        "maxSize": "5120GB",
        "protocols": {
            "simple": {
                "multipart": true,
                "path": "/upload/drive/v2/files"
            },
            "resumable": {
                "multipart": true,
                "path": "/resumable/upload/drive/v2/files"
            }
        }
    },
    "supportsSubscription": true
}

1. Only upload: https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/uploads/uploadData.

{
    "id": "analytics.management.uploads.uploadData",
    "path": "management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads",
    "httpMethod": "POST",
    "description": "Upload data for a custom data source.",
    "parameters": {
        ...
    },
    "parameterOrder": [
        "accountId",
        "webPropertyId",
        "customDataSourceId"
    ],
    "response": {
        "$ref": "Upload"
    },
    "scopes": [
        "https://www.googleapis.com/auth/analytics",
        "https://www.googleapis.com/auth/analytics.edit"
    ],
    "supportsMediaUpload": true,
    "mediaUpload": {
        "accept": [
            "application/octet-stream"
        ],
        "maxSize": "1GB",
        "protocols": {
            "simple": {
                "multipart": true,
                "path": "/upload/analytics/v3/management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads"
            },
            "resumable": {
                "multipart": true,
                "path": "/resumable/upload/analytics/v3/management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads"
            }
        }
    }
}

The distinction between the first and the last two is pretty obvious: the last two have supportsMediaUpload: true and a mediaUpload key with accepted mime types, max size, and alternate endpoints. Interestingly, they define separate endpoints for "resumable" uploads, which are not described in any documentation I can find.

However, I don't see a clear distinction between the second and third discovery documents. The second includes supportsSubscription: true while the third does not. It also includes a request key while the third does not. Of these two differences, the request key seems like the most likely candidate as the relevant distinction, but I'm still not convinced. Could there be a method which has both upload and non-upload endpoints and where a request object is never passed? Maybe not, but this all still seems very ambiguous.

[ryanseys]

This is a great thought. I checked all the cases that have no request object and accept media and most seem to follow the pattern that you're describing except it seems that fusiontables.table.importRows is documented in the discovery docs to not accept a request object but in the documentation itself it notes that it accepts both metadata only and media uploads.

So because of this, we can only decide what kind of upload to take at runtime based on the parameters you pass in. Unless you can find a better pattern 😄

[ryanseys]

fusiontables.table.importTable has the same issue as described too. Seems these are the only two.

[ryanseys]

Internal bug filed (b/16576736) to see if this is unexpected as what you are describing seems to make sense.

[ryanseys]

So the mimeType issue is a good thing to raise but also seems different because I used the API Explorer to create a draft with only metadata and the Content-Type of the request was application/json because metadata is always specified in a JSON format. But if you specify a body to the media, then you also need to specify a mimeType in this case equal to message/rfc822

[AndrewMagliozzi]

Hi @ryanseys you're right, the mimeType is slightly a different issue. Unfortunately, even when I try to specify the proper mimeType ('message/rfc822') in the resource object for a request with media to the GMail api for a function like users.drafts.create, I get this error:

Unknown field name: mimeType

If I don't specify mimeType, I get this error:

Media type \'application/octet-stream\' is not supported. Valid media types: [message/rfc822]

Currently, I am stumped. Thanks for your help!

[ryanseys]

@AndrewMagliozzi Yes, we have a possible fix available in #230 that you can try out.

[AndrewMagliozzi]

Thanks!

On Sat, Jul 26, 2014 at 2:03 PM, Ryan Seys notifications@github.com wrote:

@AndrewMagliozzi https://github.com/AndrewMagliozzi Yes, we have a
possible fix available in #230
#230 that you
can try out.

—
Reply to this email directly or view it on GitHub
#229 (comment)
.

[AndrewMagliozzi]

@ryanseys we just tested again.

Here is our attempt with a media object as part of the API call:

gmail.users.drafts.create({
  auth: authClient,
  userId: 'me',
  media: {
    mimeType: 'message/rfc822',
    body: email
    }
  }, function(err, response) {
  if (err) {
    console.log(err);
  } else {
    console.log(response);
  }
});

Here is the error that results:

{ errors: 
    [ { domain: 'global',
        reason: 'badContent',
        message: 'Unsupported content with type: message/rfc822' } ],
   code: 400,
message: 'Unsupported content with type: message/rfc822' }

When I execute the API call without a media object and specify the mimetype:

gmail.users.drafts.create({
  auth: authClient,
  userId: 'me',
  resource: { mimeType: 'message/rfc822' },
  message: {
    raw: base64encodedmessage
  }  
});

Here is the error that results:

{ errors: 
    [ { domain: 'global',
        reason: 'invalid',
        message: 'Unknown field name: mimeType',
        locationType: 'other',
        location: '' } ],
   code: 400,
message: 'Unknown field name: mimeType' }

When I execute the API call without a media object only:

gmail.users.drafts.create({
  auth: authClient,
  userId: 'me',
  message: {
    raw: base64encodedmessage
  }  
});

Here is the error that results:

{ errors: 
    [ { domain: 'global',
        reason: 'backendError',
        message: 'Backend Error' } ],
   code: 500,
   message: 'Backend Error' }

[AndrewMagliozzi]

Scratch that, SUCCESS!

We adjusted the syntax and added an email delegate as the last parameter in the authClient object (apparently this wasn't "optional"). Once 'me' was sufficiently defined in our API request, this syntax worked for us:

gmail.users.drafts.create({
  auth: authClient,
  userId: 'me',
  resource: {
    message: {
      raw: base64EncodedEmail
    }
  }
});

[ryanseys]

So the last two won't work because mimeType is only specified in media.mimeType for that API and your message needs to go inside resource. I'm guessing the first one isn't working because you didn't specify a resource. Can you try to specify resource: {} in the first example and see if you have any success? This might be an issue with https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L111

[ryanseys]

If by specifying an empty resource object we have success, then I will need to remove the check at line 111 in the previous comment.

[atrigent]

As @AndrewMagliozzi said, we got it to work. However, regarding the need to specify an empty resource: I think what you actually need to do when no resource is specified is to not send a multi-part request.

[ryanseys]

@AndrewMagliozzi @atrigent check out #235 and test with that branch (https://github.com/ryanseys/google-api-nodejs-client/tree/refresh-before). It should not build a multipart for media only requests.

[jonathannorris]

This sounds like a similar issue I am seeing with BigQuery in V1.0, I am trying to insert a new BigQuery job (bigquery.jobs.insert), and updated the formatting of my request to move the body data from the second parameter to adding it as a resource (also using the global auth object, thanks for that one :).

// NEW
params = {
    projectId: "",
    resource: body
};
bigquery.jobs.insert(params, function(err, result) { });

// OLD
googleapis.bigquery.jobs.insert(params, body).withAuthClient(oauth2).execute(..);

but I am now seeing this error from BigQuery now:

{ errors: 
   [ { domain: 'global',
       reason: 'invalid',
       message: 'Uploads are only allowed with load jobs' } ],
  code: 400,
  message: 'Uploads are only allowed with load jobs' }

Looking through the source code bigquery.jobs.insert is being marked as isMedia: true, even if I don't have any media, which I think might be causing the issue.

[ryanseys]

Just released 1.0.3 which addresses this issue.

https://github.com/google/google-api-nodejs-client/releases/tag/1.0.3

[AndrewMagliozzi]

Thanks again, Ryan. It works great now!

On Wed, Jul 30, 2014 at 7:57 PM, Ryan Seys notifications@github.com wrote:

Closed #229
#229.

—
Reply to this email directly or view it on GitHub
#229 (comment)
.