Repository navigation
report: implementing SDL on AWS/Lambda #52
Description
Structured Data Linter on AWS Lambda
Challenge 1:
Ruby creates native extensions written in C.
How to build native extensions requires compilation of the C code into the platform and environment specific machine language code?
Solution 1:
- We compiled the extensions on the same environment as the AWS/Lambda machine.
- We used
lambci/lambda: build-ruby2.5version docker image -- the same environment as used by AWS.
Challenge 2:
How to synchronize AWS Ruby environment version with SDL Ruby environment version?
Solution 2:
- We synchronized our AWS version with SDL Ruby Gemfile.
Challenge 3:
AWS Lambda only supports Deployment unzipped size of 250 MB.
See Lambda payload limits
Our deployment package is ≥ 335 MB.
How to fit the SDL deployment package into AWS/Lambda?
Solution 3:
- We removed several items to conform to AWS/Lambda constraints.
Challenge 4:
AWS/Lambda limits Response/Request payload size
See Lambda payload limits
Solution 4:
- We used AWS spot server and modified SDL code to support .zip file in "Linter By Upload" option.
Challenge 5:
How to post or get request to API Gateway endpoint and successfully run the get/post request?
Solution 5:
- We modified API Gateway settings to add a
POSTmethod. - We modified
application.jslink to point to a different location. - We modified
self linktoAPI Gateway endpoint link.
Challenge 6
You recently revised SDL. The new size is approximately 400MB.
Solution 6 (pending)
We need your guidance for removing libraries or other compaction techniques to reduce the new version to 250 MB.
Challenge 7
How to setup API Gateway Endpoint such that it can be invoked just from a specific website?
Solution (pending)
- Working on it.
Follow up
If you have private questions or suggestions please contact me at ankitadhandha@gmail.com
ankita
Activity
gkellogg commentedon Dec 23, 2019
Those extensions are going to principally include nokogiri and nokogumbo for processing RDFa, RDF/XML and Microdata. The gems will work without it (with some performance impact), and you may not need to support these serializations at all.
The hard requirement for nokogiri and nokogumo comes from the linkeddata gem (via sinatra/linkeddata); you might consider a version of the linter that doesn't use that gem, but just pulls in specific gems you need.
Perhaps we could consider versions of rack-linkeddata and sinatra-linkeddata which don't have a hard require of "linkeddata", so that you could better control this and still get the Rack/Sinatra support. Perhaps easiest would be to predicate the "require 'linkeddata'" line in rack-linkeddata to either allow the gem load to fail, or to use an environment variable.
See above.
A PR to add such support to SDL might ease integration.
A PR to add such support to SDL might ease integration.
See above.