Create a SageMaker inference endpoint with customized mannequin & prolonged container

Amazon SageMaker supplies a seamless expertise for constructing, coaching, and deploying machine studying (ML) fashions at scale. Though SageMaker presents a variety of built-in algorithms and pre-trained fashions via Amazon SageMaker JumpStart, there are situations the place you may have to convey your personal customized mannequin or use particular software program dependencies not accessible in SageMaker managed container photographs. Examples for this might embrace use circumstances like geospatial evaluation, bioinformatics analysis, or quantum machine studying. In such circumstances, SageMaker lets you lengthen its performance by creating customized container photographs and defining customized mannequin definitions. This strategy lets you package deal your mannequin artifacts, dependencies, and inference code right into a container picture, which you’ll be able to deploy as a SageMaker endpoint for real-time inference. This publish walks you thru the end-to-end strategy of deploying a single customized mannequin on SageMaker utilizing NASA’s Prithvi mannequin. The Prithvi mannequin is a first-of-its-kind temporal Imaginative and prescient transformer pre-trained by the IBM and NASA crew on contiguous US Harmonised Landsat Sentinel 2 (HLS) information. It may be finetuned for picture segmentation utilizing the mmsegmentation library to be used circumstances like burn scars detection, flood mapping, and multi-temporal crop classification. Resulting from its distinctive structure and fine-tuning dependency on the MMCV library, it’s an efficient instance of deploy advanced customized fashions to SageMaker. We exhibit use the pliability of SageMaker to deploy your personal customized mannequin, tailor-made to your particular use case and necessities. Whether or not you’re working with distinctive mannequin architectures, specialised libraries, or particular software program variations, this strategy empowers you to harness the scalability and administration capabilities of SageMaker whereas sustaining management over your mannequin’s setting and dependencies.

Answer overview

To run a customized mannequin that wants distinctive packages as a SageMaker endpoint, you might want to comply with these steps:

1. In case your mannequin requires extra packages or package deal variations unavailable from the SageMaker managed container photographs, you will want to increase one of many container photographs. By extending a SageMaker managed container vs. creating one from scratch, you may focus in your particular use case and mannequin growth as an alternative of the container infrastructure.
2. Write a Python mannequin definition utilizing the SageMaker inference.py file format.
3. Outline your mannequin artifacts and inference file inside a particular file construction, archive your mannequin recordsdata as a tar.gz file, and add your recordsdata to Amazon Easy Storage Service (Amazon S3).
4. Along with your mannequin code and an prolonged SageMaker container, use Amazon SageMaker Studio to create a mannequin, endpoint configuration, and endpoint.
5. Question the inference endpoint to verify your mannequin is working appropriately.

The next diagram illustrates the answer structure and workflow:

Conditions

You want the next conditions earlier than you may proceed. For this publish, we use the us-east-1 AWS Area:

1. Have entry to a POSIX based mostly (Mac/Linux) system or SageMaker notebooks. This publish doesn’t cowl establishing SageMaker entry and assumes a pocket book accessible to the web. Nonetheless, this isn’t a safety finest observe and shouldn’t be completed in manufacturing. To discover ways to create a SageMaker pocket book inside a digital non-public cloud (VPC), see Hook up with SageMaker AI Inside your VPC.
2. Be sure you have AWS Identification and Entry Administration (IAM) permissions for SageMaker entry; S3 bucket create, learn, and PutObject entry; AWS CodeBuild entry; Amazon Elastic Container Registry (Amazon ECR) repository entry; and the flexibility to create IAM roles.
3. Obtain the Prithvi mannequin artifacts and flood information fine-tuning:

   curl -s https://packagecloud.io/set up/repositories/github/git-lfs/script.deb.sh | sudo bash 
   sudo apt-get set up git-lfs 
   git lfs set up 
   git clone https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M 
   git clone https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-sen1floods11

Lengthen a SageMaker container picture in your mannequin

Though AWS supplies pre-built container photographs optimized for deep studying on the AWS Deep Studying Containers (DLCs) GitHub for PyTorch and TensorFlow use circumstances, there are situations the place fashions require extra libraries not included in these containers. The set up of those dependencies can take minutes or hours, so it’s extra environment friendly to pre-build these dependencies right into a customized container picture. For this instance, we deploy the Prithvi mannequin, which depends on the MMCV library for superior laptop imaginative and prescient methods. This library just isn’t accessible inside any of the SageMaker DLCs, so you’ll have to create an prolonged container so as to add it. Each MMCV and Prithvi are third-party fashions which haven’t undergone AWS safety evaluations, so please evaluate these fashions your self or use at your personal danger. This publish makes use of CodeBuild and a Docker Dockerfile to construct the prolonged container.

Full the next steps:

1. CodeBuild requires a supply location containing the supply code. Create an S3 bucket to function this supply location utilizing the next instructions:

   # generate a singular postfix 
   BUCKET_POSTFIX=$(python3 -S -c "import uuid; print(str(uuid.uuid4().hex)[:10])") 
   echo "export BUCKET_POSTFIX=${BUCKET_POSTFIX}" >> ~/.bashrc 
   echo "Your bucket identify will probably be customsagemakercontainer-codebuildsource-${BUCKET_POSTFIX}" 
   # make your bucket 
   aws s3 mb s3://customsagemakercontainer-codebuildsource-${BUCKET_POSTFIX}

2. Create an ECR repository to retailer the customized container picture produced by the CodeBuild venture. Document the repository URI as an setting variable.

   CONTAINER_REPOSITORY_NAME="prithvi" 
   aws ecr create-repository --repository-name ${CONTAINER_REPOSITORY_NAME} 
   export REPOSITORY_URI=$(aws ecr describe-repositories --repository-names ${CONTAINER_REPOSITORY_NAME} --query 'repositories[0].repositoryUri' --output textual content)

3. Create a Dockerfile for the customized container. You employ an AWS Deep Studying SageMaker framework container as the bottom picture as a result of it contains required dependencies equivalent to SageMaker libraries, PyTorch, and CUDA.

This Docker container installs the Prithvi mannequin and MMCV v1.6.2. These fashions are third-party fashions not produced by AWS and due to this fact might have safety vulnerabilities. Use at your personal danger.

4. Create a buildspec file to outline the construct course of for the CodeBuild venture. This buildspec file will instruct CodeBuild to put in the nvidia-container-toolkit to verify the Docker container has GPU entry, run the Dockerfile construct, and push the constructed container picture to your ECR repository.

   cat > buildspec.yml  /and many others/docker/daemon.json 

5. Zip and add the Dockerfile and buildspec.yml recordsdata to the S3 bucket. This zip file will function the supply code for the CodeBuild venture.
   • To put in zip on a SageMaker pocket book, run the next command:
   • With zip put in, run the next command:

     zip prithvi_container_source.zip Dockerfile buildspec.yml
     aws s3 cp prithvi_container_source.zip s3://customsagemakercontainer-codebuildsource-${BUCKET_POSTFIX}/

6. Create a CodeBuild service function so CodeBuild can entry the required AWS providers for the construct.
   • First, create a file defining the function’s belief coverage:
   • Create a file defining the service function’s permissions. This function has just a few wildcard permissions (/* or *). These may give extra permissions than wanted and break the rule of least privilege. For extra details about defining least privilege for manufacturing use circumstances, see Grant least privilege.

     cat > put-role-policy.json 

   • Create the CodeBuild service function:

     aws iam create-role --role-name CodeBuildServiceRole --assume-role-policy-document file://create-role.json

7. Seize the identify of the function Amazon Useful resource Identify (ARN) from the CLI command response and report as an setting variable:

   export CODEBUILD_SERVICE_ROLE_ARN=$(aws iam get-role --role-name CodeBuildServiceRole --query 'Function.Arn' --output textual content)

8. Connect the permission insurance policies to the service function:

   aws iam put-role-policy --role-name CodeBuildServiceRole --policy-name CodeBuildServiceRolePolicy --policy-document file://put-role-policy.json

9. Outline the configurations for the CodeBuild construct venture utilizing the construct venture JSON specification:

   cat > codebuild-project.json 

10. Create the CodeBuild venture utilizing the codebuild-project.json specification outlined within the earlier step:

    aws codebuild create-project --cli-input-json file://codebuild-project.json

11. Run a construct for the CodeBuild venture:

    aws codebuild start-build --project-name prithvi-container-build

The construct will take roughly half-hour to finish and value roughly $1.50 to run. The CodeBuild compute occasion sort gpu1.small prices $0.05 per minute.

After you run the previous command, you may press Ctrl+C to exit and run future instructions. The construct will already be working on AWS and won’t be canceled by closing the command.

12. Monitor the standing of the construct utilizing the next command and wait till you observe buildStatus=SUCCEEDED earlier than continuing to the subsequent step:

    export BUILD_ID=$(aws codebuild list-builds-for-project --project-name prithvi-container-build --query 'ids[0]' --output textual content) aws codebuild batch-get-builds --ids ${BUILD_ID} | grep buildStatus

After your CodeBuild venture has accomplished, just remember to don’t shut your terminal. The setting variables right here will probably be used once more.

Construct your inference.py file

To run a customized mannequin for inference on AWS, you might want to construct out an inference.py file that initializes your mannequin, defines the enter and output construction, and produces your inference outcomes. On this file, you will need to outline 4 capabilities:

• model_fn – Initializes your mannequin
•  input_fn – Defines how your information needs to be enter and convert to a usable format
• predict_fn – Takes the enter information and receives the prediction
• output_fn – Converts the prediction into an API name format

We use the next accomplished inference.py file for the SageMaker endpoint on this publish. Obtain this inference.py to proceed as a result of it contains the helper capabilities to course of the TIFF recordsdata wanted for this mannequin’s enter. The next code is contained throughout the inference.py and is barely proven to offer an evidence of what’s being completed within the file.

model_fn

The model_fn operate builds your mannequin, which is named and used throughout the predict_fn operate. This operate masses the mannequin weights right into a torch mannequin checkpoint, opens the mannequin config, defines international variables, instantiates the mannequin, masses the mannequin checkpoint into the mannequin, and returns the mannequin.

def model_fn(model_dir):
    # implement customized code to load the mannequin
    # load weights
    weights_path = "./code/prithvi/Prithvi_100M.pt"
    checkpoint = torch.load(weights_path, map_location="cpu")

    # learn mannequin config
    model_cfg_path = "./code/prithvi/Prithvi_100M_config.yaml"
    with open(model_cfg_path) as f:
        model_config = yaml.safe_load(f)

    model_args, train_args = model_config["model_args"], model_config["train_params"]

    international means
    international stds
    means = np.array(train_args["data_mean"]).reshape(-1, 1, 1)
    stds = np.array(train_args["data_std"]).reshape(-1, 1, 1)
    # allow us to use only one body for now (the mannequin was educated on 3 frames)
    model_args["num_frames"] = 1

    # instantiate mannequin
    mannequin = MaskedAutoencoderViT(**model_args)
    mannequin.eval()

    # load weights into mannequin
    # strict=false as a result of we're loading with only one body, however the warning is predicted
    del checkpoint['pos_embed']
    del checkpoint['decoder_pos_embed']
    _ = mannequin.load_state_dict(checkpoint, strict=False)
    
    return mannequin

input_fn

This operate defines the anticipated enter for the mannequin and load the enter to be used in predict_fn. The endpoint expects a string URL path linked to a TIFF file yow will discover on-line from the Prithvi demo on Hugging Face. This operate additionally defines the content material sort of the request despatched within the physique (equivalent to utility/json, picture/tiff).

def input_fn(input_data, content_type):
    # decode the enter information  (e.g. JSON string -> dict)
    # statistics used to normalize photographs earlier than passing to the mannequin
    raster_data = load_raster(input_data, crop=(224, 224))
    return raster_data

predict_fn:

In predict_fn, you create the prediction from the given enter. On this case, creating the prediction picture makes use of two helper capabilities particular to this endpoint (preprocess_image and enhance_raster_for_visualization). Yow will discover each capabilities right here. The preprocess_image operate normalizes the picture, then the operate makes use of torch.no_grad to disable gradient calculations for the mannequin. That is helpful throughout inference to lower inference time and scale back reminiscence utilization. Subsequent, the operate collects the prediction from the instantiated mannequin. The masks ratio determines the variety of pixels on the picture zeroed out throughout inference. The 2 unpatchify capabilities convert the smaller patchified outcomes produced by the mannequin again to the unique picture area. The operate normalized.clone() clones the normalized photographs and replaces the masked Areas from rec_img with the Areas from the pred_img. Lastly, the operate reshapes the picture again into TIFF format, removes the normalization, and returns the picture in raster format, which is effective for visualization. The results of that is a picture that may be transformed to bytes for the consumer after which visualized on the consumer’s display screen.

def predict_fn(information, mannequin):
    normalized = preprocess_image(information)
    with torch.no_grad():
        mask_ratio = 0.5
        _, pred, masks = mannequin(normalized, mask_ratio=mask_ratio)
        mask_img = mannequin.unpatchify(masks.unsqueeze(-1).repeat(1, 1, pred.form[-1])).detach().cpu()
        pred_img = mannequin.unpatchify(pred).detach().cpu()#.numpy()
        rec_img = normalized.clone()
        rec_img[mask_img == 1] = pred_img[mask_img == 1]
        rec_img_np = (rec_img.numpy().reshape(6, 224, 224) * stds) + means
        print(rec_img_np.form)
        return enhance_raster_for_visualization(rec_img_np, ref_img=information)

output_fn

output_fn returns the TIFF picture obtained from predict_fn as an array of bytes.

def output_fn(prediction, settle for):
    print(prediction.form)
    return prediction.tobytes()

Check your inference.py file

Now that you’ve downloaded the entire inference.py file, there are two choices to check your mannequin earlier than compressing the recordsdata and importing them to Amazon S3:

• Check the inference.py capabilities in your customized container inside an Amazon Elastic Compute Cloud (Amazon EC2) occasion
• Check your endpoint on a neighborhood mode  SageMaker endpoint (requires a GPU or GPU-based workspace for this mannequin)

Mannequin file construction, tar.gz compressing, and S3 add

Earlier than you begin this step, obtain the Prithvi mannequin artifacts and the Prithvi flood fine-tuning of the mannequin. The primary hyperlink will present the entire mannequin information from the bottom Prithvi mannequin, and the flood fine-tuning of the mannequin builds upon the mannequin to carry out flood plain detection on satellite tv for pc photographs. Set up git-lfs utilizing brew on Mac or utilizing https://git-lfs.com/ on Home windows to put in the GitHub repo’s giant recordsdata.

curl -s https://packagecloud.io/set up/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get set up git-lfs
git lfs set up

git clone https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M
cd Prithvi-100M
git lfs pull
git checkout c2435efd8f92329d3ca79fc8a7a70e21e675a650
git clone https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-sen1floods11
cd Prithvi-100M-sen1floods11
git lfs pull
git checkout d48937bf588cd506dd73bc4deca543446ca5530d

To create a SageMaker mannequin on the SageMaker console, you will need to retailer your mannequin information inside Amazon S3 as a result of your SageMaker endpoint will pull your mannequin artifacts immediately from Amazon S3 utilizing a tar.gz format. Inside your tar.gz file, the info should have a particular file format outlined by SageMaker. The next is the file construction for the Prithvi basis mannequin (our necessities are put in on the container, so necessities.txt has been left deliberately clean):

./mannequin
./mannequin/code/inference.py
./mannequin/code/sen1floods11_Prithvi_100M.py (prolonged mannequin config)
./mannequin/code/sen1floods11_Prithvi_100M.pth (prolonged mannequin weights)
./mannequin/code/necessities.txt
./mannequin/code/prithvi/Prithvi_100M.pt (prolonged mannequin weights)
./mannequin/code/prithvi/Prithvi_100M_config.yaml (mannequin config)
./mannequin/code/prithvi/Prithvi.py (mannequin)

This folder construction stays true for different fashions as nicely. The /code folder should maintain the inference.py file and any recordsdata used inside inference.py. These extra recordsdata are typically mannequin artifacts (configs, weights, and so forth). In our case, this would be the complete Prithvi base mannequin folder in addition to the weights and configs for the fine-tuned model we’ll use. As a result of we’ve got already put in these packages inside our container, this isn’t used; nevertheless, there nonetheless should be a necessities.txt file, in any other case your endpoint will fail to construct. All different recordsdata belong within the root folder.

With the previous file construction in place, open your terminal and route into the mannequin folder.

1. Run the next command in your terminal:

   tar -czvf mannequin.tar.gz ./

   The command will create a compressed model of your mannequin recordsdata known as mannequin.tar.gz from the recordsdata in your present listing. Now you can add this file into an S3 bucket.

2. If utilizing SageMaker, run the next command:

   sudo apt-get set up uuid-runtime

3. Now create a brand new S3 bucket. The next CLI instructions create an S3 bucket and add your mannequin.tar.gz file:

   # generate a singular postfix
   
   BUCKET_POSTFIX=$(uuidgen --random | lower -d'-' -f1)
   echo "export BUCKET_POSTFIX=${BUCKET_POSTFIX}" >> ~/.bashrc
   echo "Your bucket identify will probably be customsagemakercontainer-model-${BUCKET_POSTFIX}"
   
   # make your bucket
   aws s3 mb s3://customsagemakercontainer-model-${BUCKET_POSTFIX}
   
   # add to your bucket
   aws s3 cp mannequin.tar.gz s3://customsagemakercontainer-model-${BUCKET_POSTFIX}/mannequin.tar.gz 

The file you uploaded will probably be used within the subsequent step to outline the mannequin to be created within the endpoint.

Create SageMaker mannequin, SageMaker endpoint configuration, and SageMaker endpoint

You now create a SageMaker inference endpoint utilizing the CLI. There are three steps to making a SageMaker endpoint: create a mannequin, create an endpoint configuration, and create an endpoint.

On this publish, you’ll create a public SageMaker endpoint as a result of it will simplify working and testing the endpoint. For particulars about restrict entry to SageMaker endpoints, check with Deploy fashions with SageMaker Studio.

Full the next steps:

1. Get the ECR repository’s ARN:

   export REPOSITORY_ARN=$(aws ecr describe-repositories --repository-names ${CONTAINER_REPOSITORY_NAME} --query 'repositories[0].repositoryArn' --output textual content)

2. Create a task for the SageMaker service to imagine. Create a file defining the function’s belief coverage.

   cat > create-sagemaker-role.json 

3. Create a file defining the service function’s permissions:

   cat > put-sagemaker-role-policy.json 

4. Create the SageMaker service function:

   aws iam create-role --role-name SageMakerInferenceRole --assume-role-policy-document file://create-sagemaker-role.json
   
   export SAGEMAKER_INFERENCE_ROLE_ARN=$(aws iam get-role --role-name SageMakerInferenceRole --query 'Function.Arn' --output textual content)

5. Connect the permission insurance policies to the service function:

   aws iam put-role-policy --role-name SageMakerInferenceRole --policy-name SageMakerInferenceServiceRolePolicy --policy-document file://put-sagemaker-role-policy.json

   The mannequin definition will embrace the function you created, the ECR container picture, and the Amazon S3 location of the mannequin.tar.gz file that you just created beforehand.

6. Create a JSON file that defines the mannequin and run the create-model command:

   aws sagemaker create-model --cli-input-json file://create_model.json

   A SageMaker endpoint configuration specifies the infrastructure that the mannequin will probably be hosted on. The mannequin will probably be hosted on a ml.g4dn.xlarge occasion for GPU-based acceleration.

7. Create the endpoint configuration JSON file and create the SageMaker endpoint configuration:

   cat > create_endpoint_config.json 

8. Create the SageMaker endpoint by referencing the endpoint configuration created within the earlier step:

   aws sagemaker create-endpoint --endpoint-name prithvi-endpoint --endpoint-config-name prithvi-endpoint-config

   The ml.g4dn.xlarge inference endpoint will value $0.736 per hour whereas working. It’s going to take a number of minutes for the endpoint to complete deploying.

9. Verify the standing utilizing the next command, ready for it to return InService:

   aws sagemaker describe-endpoint --endpoint-name prithvi-endpoint --query "EndpointStatus" --output textual content

When the endpoint’s standing is InService, proceed to the subsequent part.

Check your customized SageMaker inference endpoint

To check your SageMaker endpoint, you’ll question your endpoint with a picture and show it. The next command sends a URL that references a TIFF picture to the SageMaker endpoint, the mannequin sends again a byte array, and the command reforms the byte array into a picture. Open up a pocket book regionally or on Sagemaker Studio JupyterLab. The beneath code will must be run outdoors of the command line to view the picture

from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONSerializer

payload = "https://huggingface.co/ibm-nasa-geospatial/Prithvi-EO-1.0-100M/resolve/major/examples/HLS.L30.T13REN.2018013T172747.v2.0.B02.B03.B04.B05.B06.B07_cropped.tif"

predictor = Predictor(endpoint_name="prithvi-endpoint")
predictor.serializer = JSONSerializer()

predictions = predictor.predict(payload)

This Python code creates a predictor object in your endpoint and units the predictor’s serializer to NumPy for the conversion on the endpoint. It queries the predictor object utilizing a payload of a URL pointing to a TIFF picture. You employ a helper operate to show the picture and improve the raster. It is possible for you to to seek out that helper operate right here. After you add the helper operate, show the picture:

import numpy as np
import matplotlib.pyplot as plt

NO_DATA_FLOAT = 0.0001
PERCENTILES = (0.1, 99.9)
NO_DATA = -9999

nppred = np.frombuffer(predictions).reshape((224, 224, 3))

def enhance_raster_for_visualization(raster, ref_img=None):
    if ref_img is None:
        ref_img = raster
    channels = []
    for channel in vary(raster.form[0]):
        valid_mask = np.ones_like(ref_img[channel], dtype=bool)
        valid_mask[ref_img[channel] == NO_DATA_FLOAT] = False
        minutes, maxs = np.percentile(ref_img[channel][valid_mask], PERCENTILES)
        normalized_raster = (raster[channel] - minutes) / (maxs - minutes)
        normalized_raster[~valid_mask] = 0
        clipped = np.clip(normalized_raster, 0, 1)
        channels.append(clipped)
    clipped = np.stack(channels)
    channels_last = np.moveaxis(clipped, 0, -1)[..., :3]
    rgb = channels_last[..., ::-1]
    return rgb
    
raster_for_visualization = enhance_raster_for_visualization(nppred)
plt.imshow(nppred)
plt.present()

It is best to observe a picture that has been taken from a satellite tv for pc.

Clear up

To wash up the assets from this publish and keep away from incurring prices, comply with these steps:

1. Delete the SageMaker endpoint, endpoint configuration, and mannequin.
2. Delete the ECR picture and repository.
3. Delete the mannequin.tar.gz file within the S3 bucket that was created.
4. Delete the customsagemakercontainer-model and customsagemakercontainer-codebuildsource S3 buckets.

Conclusion

On this publish, we prolonged a SageMaker container to incorporate customized dependencies, wrote a Python script to run a customized ML mannequin, and deployed that mannequin on the SageMaker container inside a SageMaker endpoint for real-time inference. This answer produces a working GPU-enabled endpoint for inference queries. You need to use this identical course of to create customized mannequin SageMaker endpoints by extending different SageMaker containers and writing an inference.py file for brand spanking new customized fashions. Moreover, with changes, you possibly can create a multi-model SageMaker endpoint for customized fashions or run a batch processing endpoint for situations the place you run giant batches of queries directly. These options allow you to transcend the preferred fashions used right now and customise fashions to suit your personal distinctive use case.

In regards to the Authors

Aidan is a options architect supporting US federal authorities well being clients. He assists clients by growing technical architectures and offering finest practices on Amazon Internet Providers (AWS) cloud with a give attention to AI/ML providers. In his free time, Aidan enjoys touring, lifting, and cooking

Nate is a options architect supporting US federal authorities sciences clients. He assists clients in growing technical architectures on Amazon Internet Providers (AWS), with a give attention to information analytics and excessive efficiency computing. In his free time, he enjoys snowboarding and {golfing}.

Charlotte is a options architect on the aerospace & satellite tv for pc crew at Amazon Internet Providers (AWS), the place she helps clients obtain their mission aims via modern cloud options. Charlotte focuses on machine studying with a give attention to Generative AI. In her free time, she enjoys touring, portray, and working.