We are planning to use octopus deploy for Iaac and code deployment in AWS cloud. However we want to use aws autoscaling concept also to scale up and scale down. I want to know if there is any automated mechanism to register and de-register target in octopus deploy in case of scaling up of scaling down a EC2 instance.
You have a few options.
With AWS, you can configure an AWS lambda to run when a machine is remove from an autoscaling group. https://aws.amazon.com/blogs/compute/using-aws-lambda-with-auto-scaling-lifecycle-hooks/. The Lambda could call an API call in Octopus to delete the machine.
Or, you can configure your machine health checks in Octopus to automatically remove unhealthy machines.
I would take a hard look at the AWS Lambda function.
See this KB article for more details: Using Octopus Deploy AWS Auto-Scaling Groups