import os
import requests
import time
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

GITHUB_ACCESS_TOKEN = os.getenv("GITHUB_ACCESS_TOKEN")

def fetch_repo_contents(owner, repo):
    """
    Fetch the contents of a GitHub repository.
    """
    url = f"https://api.github.com/repos/{owner}/{repo}/contents"
    headers = {"Authorization": f"token {GITHUB_ACCESS_TOKEN}"}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        raise Exception(f"Failed to fetch repository contents: {response.status_code}")

def filter_python_files(contents):
    """
    Filter out Python files from the repository contents.
    """
    return [file for file in contents if file["name"].endswith(".py")]

def download_file_contents(download_url):
    """
    Download the raw content of a file from GitHub.
    """
    headers = {"Authorization": f"token {GITHUB_ACCESS_TOKEN}"}
    response = requests.get(download_url, headers=headers)
    if response.status_code == 200:
        return response.text
    else:
        raise Exception(f"Failed to download file: {response.status_code}")

def make_github_request(url):
    """
    Make a GitHub API request with rate limit handling.
    """
    headers = {"Authorization": f"token {GITHUB_ACCESS_TOKEN}"}
    while True:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return response.json()
        elif response.status_code == 403 and "rate limit" in response.text:
            reset_time = int(response.headers["X-RateLimit-Reset"])
            sleep_time = max(reset_time - time.time(), 0) + 1  # Add 1 second buffer
            print(f"Rate limit exceeded. Sleeping for {sleep_time} seconds.")
            time.sleep(sleep_time)
        else:
            raise Exception(f"Failed to make request: {response.status_code}")

This script interacts with the GitHub API to fetch, filter, and download Python files from a given repository while handling rate limits.

1. Load Environment Variables
   The script imports necessary modules (os, requests, time, and dotenv). It then calls load_dotenv() to load environment variables from a .env file. It retrieves the GitHub access token from the environment using os.getenv("GITHUB_ACCESS_TOKEN").
2. Fetch Repository Contents
   The fetch_repo_contents(owner, repo) function constructs a GitHub API URL for fetching the repository’s file contents. It includes an authorization header with the access token. If GitHub returns a successful response (200), the function parses and returns the JSON response. Otherwise, it raises an exception with the status code.
3. Filter Python Files
   The filter_python_files(contents) function iterates through the repository contents and selects only files ending in .py, returning a list of Python files.
4. Download File Contents
   The download_file_contents(download_url) function makes an authenticated request to download a file’s raw content from GitHub. If the request succeeds (200), it returns the file’s text content. Otherwise, it raises an exception.
5. Handle API Rate Limits
   The make_github_request(url) function makes a request to the GitHub API while handling rate limits. If the request succeeds (200), it returns the JSON response. If the request fails due to rate limiting (403), it calculates the wait time using the X-RateLimit-Reset header and pauses execution before retrying. If the request fails for other reasons, it raises an exception.

This script ensures secure API access, filters relevant files, and handles GitHub’s rate limits efficiently.

Step 3: Update routes.py

Add a new route to fetch and display repository contents.

Update: app/routes.py to contain the following code:

from flask import Blueprint, jsonify, request
from .utils.docstring_generator import generate_docstring
from app.utils.github_api import fetch_repo_contents, filter_python_files, download_file_contents

main_bp = Blueprint('main', __name__)

@main_bp.route('/')
def home():
    return "Welcome to the AI-Powered Documentation Assistant!"

@main_bp.route('/generate-docstring', methods=['POST'])

def generate_docstring_route():
    code = request.json.get('code')
    if not code:
        return jsonify({"error": "No code provided"}), 400

    try:
        docstring = generate_docstring(code)
        return jsonify({"docstring": docstring})
    except Exception as e:
        return jsonify({"error": str(e)}), 500
    
@main_bp.route("/fetch-repo", methods=["POST"])
def fetch_repo():
    """
    Fetch and display Python files from a GitHub repository.
    """
    data = request.json
    owner = data.get("owner")
    repo = data.get("repo")

    try:
        contents = fetch_repo_contents(owner, repo)
        python_files = filter_python_files(contents)
        return jsonify({"python_files": python_files})
    except Exception as e:
        return jsonify({"error": str(e)}), 500

The fetch_repo() function retrieves Python files from a specified GitHub repository and returns them as a JSON response.

1. Receive Request Data
   The function extracts the repository owner and name from the incoming JSON request using request.json.get("owner") and request.json.get("repo").
2. Fetch Repository Contents
   It calls fetch_repo_contents(owner, repo) to retrieve the contents of the specified GitHub repository.
3. Filter Python Files
   It processes the fetched repository contents using filter_python_files(contents), which selects only files ending in .py.
4. Return Response
   If successful, the function returns a JSON response containing the list of Python files. If an error occurs, it catches the exception and returns an error message with a 500 status code.

This function enables the Flask app to interact with the GitHub API and extract Python files from repositories dynamically.

Lesson 4: Parsing Code for Documentation

Objective

In this lesson, you’ll use Python’s Abstract Syntax Tree (AST) to analyze code, extract metadata, and handle common parsing errors.

Step 1: Update code_parser.py

Add logic to parse Python code and extract functions, classes, and metadata.

Update: app/utils/code_parser.py to contain the following code:

import ast

def parse_code(code):
    tree = ast.parse(code)
    functions = [node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)]
    classes = [node for node in ast.walk(tree) if isinstance(node, ast.ClassDef)]
    return {"functions": functions, "classes": classes}

def extract_functions_and_classes(code):
    """
    Extract functions and classes from Python code using AST.
    """
    tree = ast.parse(code)
    functions = [node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)]
    classes = [node for node in ast.walk(tree) if isinstance(node, ast.ClassDef)]
    return functions, classes

def extract_function_signature(func_node):
    """
    Extract function signature (name, args, returns).
    """
    args = [arg.arg for arg in func_node.args.args]
    returns = ast.unparse(func_node.returns) if func_node.returns else None
    return {
        "name": func_node.name,
        "args": args,
        "returns": returns
    }

def extract_class_metadata(class_node):
    """
    Extract class metadata (name, methods, docstring).
    """
    methods = [node.name for node in ast.walk(class_node) if isinstance(node, ast.FunctionDef)]
    docstring = ast.get_docstring(class_node)
    return {
        "name": class_node.name,
        "methods": methods,
        "docstring": docstring
    }

This script analyzes Python code using the Abstract Syntax Tree (AST) module to extract functions, classes, and their metadata.

1. Parse Code and Identify Functions and Classes
   • The parse_code(code) function parses the given Python code into an AST.
   • It walks through the AST tree to collect function definitions (ast.FunctionDef) and class definitions (ast.ClassDef).
   • It returns a dictionary containing lists of functions and classes.
2. Extract Functions and Classes
   • The extract_functions_and_classes(code) function also parses the given Python code into an AST.
   • It extracts function and class definitions separately and returns them as two lists.
3. Extract Function Signature
   • The extract_function_signature(func_node) function retrieves the function name, its arguments, and return type.
   • It extracts argument names from func_node.args.args.
   • If the function has a return type annotation, it un-parses it using ast.unparse(func_node.returns).
   • It returns a dictionary containing the function name, argument list, and return type.
4. Extract Class Metadata
   • The extract_class_metadata(class_node) function retrieves the class name, its methods, and its docstring.
   • It collects method names by walking through the class node and identifying FunctionDef nodes.
   • It extracts the class docstring using ast.get_docstring(class_node).
   • It returns a dictionary containing the class name, a list of methods, and the docstring.

This script enables Python code analysis by extracting structural information about functions and classes.

Step 2: Update routes.py

Add a new route to parse and display metadata from a Python file.

Update: app/routes.pyto contain the following code:

from flask import Blueprint, jsonify, request
from .utils.docstring_generator import generate_docstring
from app.utils.github_api import fetch_repo_contents, filter_python_files, download_file_contents
from app.utils.code_parser import extract_functions_and_classes, extract_function_signature, extract_class_metadata


main_bp = Blueprint('main', __name__)

@main_bp.route('/')
def home():
    return "Welcome to the AI-Powered Documentation Assistant!"

@main_bp.route('/generate-docstring', methods=['POST'])

def generate_docstring_route():
    code = request.json.get('code')
    if not code:
        return jsonify({"error": "No code provided"}), 400

    try:
        docstring = generate_docstring(code)
        return jsonify({"docstring": docstring})
    except Exception as e:
        return jsonify({"error": str(e)}), 500
    
@main_bp.route("/fetch-repo", methods=["POST"])
def fetch_repo():
    """
    Fetch and display Python files from a GitHub repository.
    """
    data = request.json
    owner = data.get("owner")
    repo = data.get("repo")

    try:
        contents = fetch_repo_contents(owner, repo)
        python_files = filter_python_files(contents)
        return jsonify({"python_files": python_files})
    except Exception as e:
        return jsonify({"error": str(e)}), 500

@main_bp.route("/parse-file", methods=["POST"])
def parse_file():
    """
    Parse a Python file and extract metadata.
    """
    data = request.json
    download_url = data.get("download_url")

    try:
        code = download_file_contents(download_url)
        functions, classes = extract_functions_and_classes(code)
        function_metadata = [extract_function_signature(func) for func in functions]
        class_metadata = [extract_class_metadata(cls) for cls in classes]
        return jsonify({
            "functions": function_metadata,
            "classes": class_metadata
        })
    except Exception as e:
        return jsonify({"error": str(e)}), 500

The parse_file() function retrieves a Python file from a given URL, analyzes its contents, and extracts metadata about its functions and classes.

1. Receive Request Data
   The function reads the JSON request body and extracts the download_url parameter.
2. Download the File
   It calls download_file_contents(download_url) to fetch the raw content of the Python file.
3. Extract Functions and Classes
   It processes the file content using extract_functions_and_classes(code), which returns lists of function and class definitions.
4. Generate Function Metadata
   It iterates over the extracted functions and calls extract_function_signature(func) to retrieve each function’s name, arguments, and return type.
5. Generate Class Metadata
   It iterates over the extracted classes and calls extract_class_metadata(cls) to retrieve each class’s name, methods, and docstring.
6. Return the Metadata
   The function returns a JSON response containing the extracted function and class metadata. If an error occurs, it catches the exception and returns an error message with a 500 status code.

This function automates the process of analyzing Python files and extracting useful metadata for documentation or code analysis.