go vendor

This commit is contained in:
lealife committed 2017-11-30 19:55:33 +08:00
1 parent 2856da6888
commit 0fb92efbf3
670 files changed
+199010

No files matched your search

+12
View File
@@ -0,0 +1,12 @@
Copyright (c) 2012-2013, Martin Angers & Contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+109
View File
@@ -0,0 +1,109 @@
# goquery - a little like that j-thing, only in Go
GoQuery brings a syntax and a set of features similar to [jQuery][] to the [Go language][go]. It is based on the experimental html package and the CSS Selector library [cascadia][]. Since the experimental html parser returns tokens (nodes), and not a full-featured DOM object, jQuery's manipulation and modification functions have been left off (no point in modifying data in the parsed tree of the HTML, it has no effect).
Supported functions are query-oriented features (`hasClass()`, `attr()` and the likes), as well as traversing functions that make sense given what we have to work with. This makes GoQuery a great library for scraping web pages.
Syntax-wise, it is as close as possible to jQuery, with the same function names when possible, and that warm and fuzzy chainable interface. jQuery being the ultra-popular library that it is, I felt that writing a similar HTML-manipulating library was better to follow its API than to start anew (in the same spirit as Go's `fmt` package), even though some of its methods are less than intuitive (looking at you, [index()][index]...).
## Installation
$ go get github.com/PuerkitoBio/goquery
(optional) To run unit tests:
$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
$ go test
(optional) To run benchmarks:
$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
$ go test -bench=".*"
## Changelog
* **v0.3.2** : Add `NewDocumentFromReader()` (thanks jweir) which allows creating a goquery document from an io.Reader.
* **v0.3.1** : Add `NewDocumentFromResponse()` (thanks assassingj) which allows creating a goquery document from an http response.
* **v0.3.0** : Add `EachWithBreak()` which allows to break out of an `Each()` loop by returning false. This function was added instead of changing the existing `Each()` to avoid breaking compatibility.
* **v0.2.1** : Make go-getable, now that [go.net/html is Go1.0-compatible][gonet] (thanks to @matrixik for pointing this out).
* **v0.2.0** : Add support for negative indices in Slice(). **BREAKING CHANGE** `Document.Root` is removed, `Document` is now a `Selection` itself (a selection of one, the root element, just like `Document.Root` was before). Add jQuery's Closest() method.
* **v0.1.1** : Add benchmarks to use as baseline for refactorings, refactor Next...() and Prev...() methods to use the new html package's linked list features (Next/PrevSibling, FirstChild). Good performance boost (40+% in some cases).
* **v0.1.0** : Initial release.
## API
GoQuery exposes two classes, `Document` and `Selection`. Unlike jQuery, which is loaded as part of a DOM document, and thus acts on its containing document, GoQuery doesn't know which HTML document to act upon. So it needs to be told, and that's what the `Document` class is for. It holds the root document node as the initial Selection object to manipulate.
jQuery often has many variants for the same function (no argument, a selector string argument, a jQuery object argument, a DOM element argument, ...). Instead of exposing the same features in GoQuery as a single method with variadic empty interface arguments, I use statically-typed signatures following this naming convention:
* When the jQuery equivalent can be called with no argument, it has the same name as jQuery for the no argument signature (e.g.: `Prev()`), and the version with a selector string argument is called `XxxFiltered()` (e.g.: `PrevFiltered()`)
* When the jQuery equivalent **requires** one argument, the same name as jQuery is used for the selector string version (e.g.: `Is()`)
* The signatures accepting a jQuery object as argument are defined in GoQuery as `XxxSelection()` and take a `*Selection` object as argument (e.g.: `FilterSelection()`)
* The signatures accepting a DOM element as argument in jQuery are defined in GoQuery as `XxxNodes()` and take a variadic argument of type `*html.Node` (e.g.: `FilterNodes()`)
* Finally, the signatures accepting a function as argument in jQuery are defined in GoQuery as `XxxFunction()` and take a function as argument (e.g.: `FilterFunction()`)
GoQuery's complete [godoc reference documentation can be found here][doc].
Please note that Cascadia's selectors do NOT necessarily match all supported selectors of jQuery (Sizzle). See the [cascadia project][cascadia] for details.
## Examples
Taken from example_test.go:
```Go
import (
"fmt"
// In real use, this import would be required (not in this example, since it
// is part of the goquery package)
//"github.com/PuerkitoBio/goquery"
"strconv"
)
// This example scrapes the 10 reviews shown on the home page of MetalReview.com,
// the best metal review site on the web :) (and no, I'm not affiliated to them!)
func ExampleScrape_MetalReview() {
// Load the HTML document (in real use, the type would be *goquery.Document)
var doc *Document
var e error
if doc, e = NewDocument("http://metalreview.com"); e != nil {
panic(e.Error())
}
// Find the review items (the type of the Selection would be *goquery.Selection)
doc.Find(".slider-row:nth-child(1) .slider-item").Each(func(i int, s *Selection) {
var band, title string
var score float64
// For each item found, get the band, title and score, and print it
band = s.Find("strong").Text()
title = s.Find("em").Text()
if score, e = strconv.ParseFloat(s.Find(".score").Text(), 64); e != nil {
// Not a valid float, ignore score
fmt.Printf("Review %d: %s - %s.\n", i, band, title)
} else {
// Print all, including score
fmt.Printf("Review %d: %s - %s (%2.1f).\n", i, band, title, score)
}
})
// To see the output of the Example while running the test suite (go test), simply
// remove the leading "x" before Output on the next line. This will cause the
// example to fail (all the "real" tests should pass).
// xOutput: voluntarily fail the Example output.
}
```
## License
The [BSD 3-Clause license][bsd], the same as the [Go language][golic]. Cascadia's license is [here][caslic].
[jquery]: http://jquery.com/
[go]: http://golang.org/
[cascadia]: http://code.google.com/p/cascadia/
[bsd]: http://opensource.org/licenses/BSD-3-Clause
[golic]: http://golang.org/LICENSE
[caslic]: http://code.google.com/p/cascadia/source/browse/LICENSE
[doc]: http://godoc.org/github.com/PuerkitoBio/goquery
[index]: http://api.jquery.com/index/
[gonet]: http://code.google.com/p/go/source/detail?r=f7f5159120f51ba0070774d3c5907969b5fe7858&repo=net
+85
View File
@@ -0,0 +1,85 @@
package goquery
import (
"code.google.com/p/go.net/html"
)
// First() reduces the set of matched elements to the first in the set.
// It returns a new Selection object.
func (this *Selection) First() *Selection {
return this.Eq(0)
}
// Last() reduces the set of matched elements to the last in the set.
// It returns a new Selection object.
func (this *Selection) Last() *Selection {
return this.Eq(-1)
}
// Eq() reduces the set of matched elements to the one at the specified index.
// If a negative index is given, it counts backwards starting at the end of the
// set. It returns a new Selection object, and an empty Selection object if the
// index is invalid.
func (this *Selection) Eq(index int) *Selection {
if index < 0 {
index += len(this.Nodes)
}
return this.Slice(index, index+1)
}
// Slice() reduces the set of matched elements to a subset specified by a range
// of indices.
func (this *Selection) Slice(start int, end int) *Selection {
if start < 0 {
start += len(this.Nodes)
}
if end < 0 {
end += len(this.Nodes)
}
return pushStack(this, this.Nodes[start:end])
}
// Get() retrieves the underlying node at the specified index.
// Get() without parameter is not implemented, since the node array is available
// on the Selection object.
func (this *Selection) Get(index int) *html.Node {
if index < 0 {
index += len(this.Nodes) // Negative index gets from the end
}
return this.Nodes[index]
}
// Index() returns the position of the first element within the Selection object
// relative to its sibling elements.
func (this *Selection) Index() int {
if len(this.Nodes) > 0 {
return newSingleSelection(this.Nodes[0], this.document).PrevAll().Length()
}
return -1
}
// IndexSelector() returns the position of the first element within the
// Selection object relative to the elements matched by the selector, or -1 if
// not found.
func (this *Selection) IndexSelector(selector string) int {
if len(this.Nodes) > 0 {
sel := this.document.Find(selector)
return indexInSlice(sel.Nodes, this.Nodes[0])
}
return -1
}
// IndexOfNode() returns the position of the specified node within the Selection
// object, or -1 if not found.
func (this *Selection) IndexOfNode(node *html.Node) int {
return indexInSlice(this.Nodes, node)
}
// IndexOfSelection() returns the position of the first node in the specified
// Selection object within this Selection object, or -1 if not found.
func (this *Selection) IndexOfSelection(s *Selection) int {
if s != nil && len(s.Nodes) > 0 {
return indexInSlice(this.Nodes, s.Nodes[0])
}
return -1
}
+91
View File
@@ -0,0 +1,91 @@
// Copyright (c) 2012-2013, Martin Angers & Contributors
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation and/or
// other materials provided with the distribution.
// * Neither the name of the author nor the names of its contributors may be used to
// endorse or promote products derived from this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
// OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
// AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
// CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
// WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
// WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
/*
Package goquery implements features similar to jQuery, including the chainable
syntax, to manipulate and query an HTML document (the modification functions of jQuery are not included).
It uses Cascadia as CSS selector (similar to Sizzle for jQuery).
To provide a chainable interface, error management is strict, and goquery panics
if an invalid Cascadia selector is used (this is consistent with the behavior of
jQuery/Sizzle/document.querySelectorAll, where an error is thrown). This is
necessary since multiple return values cannot be used to allow a chainable
interface.
It is hosted on GitHub, along with additional documentation in the README.md
file: https://github.com/puerkitobio/goquery
The various methods are split into files based on the category of behavior:
* array.go : array-like positional manipulation of the selection.
- Eq()
- First()
- Get()
- Index...()
- Last()
- Slice()
* expand.go : methods that expand or augment the selection's set.
- Add...()
- AndSelf()
- Union(), which is an alias for AddSelection()
* filter.go : filtering methods, that reduce the selection's set.
- End()
- Filter...()
- Has...()
- Intersection(), which is an alias of FilterSelection()
- Not...()
* iteration.go : methods to loop over the selection's nodes.
- Each()
- EachWithBreak()
- Map()
* property.go : methods that inspect and get the node's properties values.
- Attr()
- Html()
- Length()
- Size(), which is an alias for Length()
- Text()
* query.go : methods that query, or reflect, a node's identity.
- Contains()
- HasClass()
- Is...()
* traversal.go : methods to traverse the HTML document tree.
- Children...()
- Contents()
- Find...()
- Next...()
- Parent[s]...()
- Prev...()
- Siblings...()
* type.go : definition of the types exposed by GoQuery.
- Document
- Selection
*/
package goquery
+40
View File
@@ -0,0 +1,40 @@
package goquery
import (
"code.google.com/p/go.net/html"
)
// Add() adds the selector string's matching nodes to those in the current
// selection and returns a new Selection object.
// The selector string is run in the context of the document of the current
// Selection object.
func (this *Selection) Add(selector string) *Selection {
return this.AddNodes(findWithSelector([]*html.Node{this.document.rootNode}, selector)...)
}
// AddSelection() adds the specified Selection object's nodes to those in the
// current selection and returns a new Selection object.
func (this *Selection) AddSelection(sel *Selection) *Selection {
if sel == nil {
return this.AddNodes()
}
return this.AddNodes(sel.Nodes...)
}
// Union() is an alias for AddSelection().
func (this *Selection) Union(sel *Selection) *Selection {
return this.AddSelection(sel)
}
// AddNodes() adds the specified nodes to those in the
// current selection and returns a new Selection object.
func (this *Selection) AddNodes(nodes ...*html.Node) *Selection {
return pushStack(this, appendWithoutDuplicates(this.Nodes, nodes))
}
// AndSelf() adds the previous set of elements on the stack to the current set.
// It returns a new Selection object containing the current Selection combined
// with the previous one.
func (this *Selection) AndSelf() *Selection {
return this.AddSelection(this.prevSel)
}
+141
View File
@@ -0,0 +1,141 @@
package goquery
import (
"code.google.com/p/cascadia"
"code.google.com/p/go.net/html"
)
// Filter() reduces the set of matched elements to those that match the selector string.
// It returns a new Selection object for this subset of matching elements.
func (this *Selection) Filter(selector string) *Selection {
return pushStack(this, winnow(this, selector, true))
}
// Not() removes elements from the Selection that match the selector string.
// It returns a new Selection object with the matching elements removed.
func (this *Selection) Not(selector string) *Selection {
return pushStack(this, winnow(this, selector, false))
}
// FilterFunction() reduces the set of matched elements to those that pass the function's test.
// It returns a new Selection object for this subset of elements.
func (this *Selection) FilterFunction(f func(int, *Selection) bool) *Selection {
return pushStack(this, winnowFunction(this, f, true))
}
// Not() removes elements from the Selection that pass the function's test.
// It returns a new Selection object with the matching elements removed.
func (this *Selection) NotFunction(f func(int, *Selection) bool) *Selection {
return pushStack(this, winnowFunction(this, f, false))
}
// FilterNodes() reduces the set of matched elements to those that match the specified nodes.
// It returns a new Selection object for this subset of elements.
func (this *Selection) FilterNodes(nodes ...*html.Node) *Selection {
return pushStack(this, winnowNodes(this, nodes, true))
}
// Not() removes elements from the Selection that match the specified nodes.
// It returns a new Selection object with the matching elements removed.
func (this *Selection) NotNodes(nodes ...*html.Node) *Selection {
return pushStack(this, winnowNodes(this, nodes, false))
}
// FilterSelection() reduces the set of matched elements to those that match a
// node in the specified Selection object.
// It returns a new Selection object for this subset of elements.
func (this *Selection) FilterSelection(s *Selection) *Selection {
if s == nil {
return pushStack(this, winnowNodes(this, nil, true))
}
return pushStack(this, winnowNodes(this, s.Nodes, true))
}
// Not() removes elements from the Selection that match a node in the specified
// Selection object.
// It returns a new Selection object with the matching elements removed.
func (this *Selection) NotSelection(s *Selection) *Selection {
if s == nil {
return pushStack(this, winnowNodes(this, nil, false))
}
return pushStack(this, winnowNodes(this, s.Nodes, false))
}
// Intersection() is an alias for FilterSelection().
func (this *Selection) Intersection(s *Selection) *Selection {
return this.FilterSelection(s)
}
// Has() reduces the set of matched elements to those that have a descendant
// that matches the selector.
// It returns a new Selection object with the matching elements.
func (this *Selection) Has(selector string) *Selection {
return this.HasSelection(this.document.Find(selector))
}
// HasNodes() reduces the set of matched elements to those that have a
// descendant that matches one of the nodes.
// It returns a new Selection object with the matching elements.
func (this *Selection) HasNodes(nodes ...*html.Node) *Selection {
return this.FilterFunction(func(_ int, s *Selection) bool {
// Add all nodes that contain one of the specified nodes
for _, n := range nodes {
if s.Contains(n) {
return true
}
}
return false
})
}
// HasSelection() reduces the set of matched elements to those that have a
// descendant that matches one of the nodes of the specified Selection object.
// It returns a new Selection object with the matching elements.
func (this *Selection) HasSelection(sel *Selection) *Selection {
if sel == nil {
return this.HasNodes()
}
return this.HasNodes(sel.Nodes...)
}
// End() ends the most recent filtering operation in the current chain and
// returns the set of matched elements to its previous state.
func (this *Selection) End() *Selection {
if this.prevSel != nil {
return this.prevSel
}
return newEmptySelection(this.document)
}
// Filter based on a selector string, and the indicator to keep (Filter) or
// to get rid of (Not) the matching elements.
func winnow(sel *Selection, selector string, keep bool) []*html.Node {
cs := cascadia.MustCompile(selector)
// Optimize if keep is requested
if keep {
return cs.Filter(sel.Nodes)
} else {
// Use grep
return grep(sel, func(i int, s *Selection) bool {
return !cs.Match(s.Get(0))
})
}
return nil
}
// Filter based on an array of nodes, and the indicator to keep (Filter) or
// to get rid of (Not) the matching elements.
func winnowNodes(sel *Selection, nodes []*html.Node, keep bool) []*html.Node {
return grep(sel, func(i int, s *Selection) bool {
return isInSlice(nodes, s.Get(0)) == keep
})
}
// Filter based on a function test, and the indicator to keep (Filter) or
// to get rid of (Not) the matching elements.
func winnowFunction(sel *Selection, f func(int, *Selection) bool, keep bool) []*html.Node {
return grep(sel, func(i int, s *Selection) bool {
return f(i, s) == keep
})
}
+33
View File
@@ -0,0 +1,33 @@
package goquery
// Each() iterates over a Selection object, executing a function for each
// matched element. It returns the current Selection object.
func (this *Selection) Each(f func(int, *Selection)) *Selection {
for i, n := range this.Nodes {
f(i, newSingleSelection(n, this.document))
}
return this
}
// EachWithBreak() iterates over a Selection object, executing a function for each
// matched element. It is identical to `Each()` except that it is possible to break
// out of the loop by returning `false` in the callback function. It returns the
// current Selection object.
func (this *Selection) EachWithBreak(f func(int, *Selection) bool) *Selection {
for i, n := range this.Nodes {
if !f(i, newSingleSelection(n, this.document)) {
return this
}
}
return this
}
// Map() passes each element in the current matched set through a function,
// producing a slice of string holding the returned values.
func (this *Selection) Map(f func(int, *Selection) string) (result []string) {
for i, n := range this.Nodes {
result = append(result, f(i, newSingleSelection(n, this.document)))
}
return result
}
+90
View File
@@ -0,0 +1,90 @@
package goquery
import (
"bytes"
"code.google.com/p/go.net/html"
)
// Attr() gets the specified attribute's value for the first element in the
// Selection. To get the value for each element individually, use a looping
// construct such as Each() or Map() method.
func (this *Selection) Attr(attrName string) (val string, exists bool) {
if len(this.Nodes) == 0 {
return
}
return getAttributeValue(attrName, this.Nodes[0])
}
// Text() gets the combined text contents of each element in the set of matched
// elements, including their descendants.
func (this *Selection) Text() string {
var buf bytes.Buffer
// Slightly optimized vs calling Each(): no single selection object created
for _, n := range this.Nodes {
buf.WriteString(getNodeText(n))
}
return buf.String()
}
// Size() is an alias for Length().
func (this *Selection) Size() int {
return this.Length()
}
// Length() returns the number of elements in the Selection object.
func (this *Selection) Length() int {
return len(this.Nodes)
}
// Html() gets the HTML contents of the first element in the set of matched
// elements. It includes text and comment nodes.
func (this *Selection) Html() (ret string, e error) {
// Since there is no .innerHtml, the HTML content must be re-created from
// the nodes usint html.Render().
var buf bytes.Buffer
if len(this.Nodes) > 0 {
for c := this.Nodes[0].FirstChild; c != nil; c = c.NextSibling {
e = html.Render(&buf, c)
if e != nil {
return
}
}
ret = buf.String()
}
return
}
// Get the specified node's text content.
func getNodeText(node *html.Node) string {
if node.Type == html.TextNode {
// Keep newlines and spaces, like jQuery
return node.Data
} else if node.FirstChild != nil {
var buf bytes.Buffer
for c := node.FirstChild; c != nil; c = c.NextSibling {
buf.WriteString(getNodeText(c))
}
return buf.String()
}
return ""
}
// Private function to get the specified attribute's value from a node.
func getAttributeValue(attrName string, n *html.Node) (val string, exists bool) {
if n == nil {
return
}
for _, a := range n.Attr {
if a.Key == attrName {
val = a.Val
exists = true
return
}
}
return
}
+71
View File
@@ -0,0 +1,71 @@
package goquery
import (
"code.google.com/p/cascadia"
"code.google.com/p/go.net/html"
"regexp"
"strings"
)
var rxClassTrim = regexp.MustCompile("[\t\r\n]")
// Is() checks the current matched set of elements against a selector and
// returns true if at least one of these elements matches.
func (this *Selection) Is(selector string) bool {
if len(this.Nodes) > 0 {
// Attempt a match with the selector
cs := cascadia.MustCompile(selector)
if len(this.Nodes) == 1 {
return cs.Match(this.Nodes[0])
} else {
return len(cs.Filter(this.Nodes)) > 0
}
}
return false
}
// IsFunction() checks the current matched set of elements against a predicate and
// returns true if at least one of these elements matches.
func (this *Selection) IsFunction(f func(int, *Selection) bool) bool {
return this.FilterFunction(f).Length() > 0
}
// IsSelection() checks the current matched set of elements against a Selection object
// and returns true if at least one of these elements matches.
func (this *Selection) IsSelection(s *Selection) bool {
return this.FilterSelection(s).Length() > 0
}
// IsNodes() checks the current matched set of elements against the specified nodes
// and returns true if at least one of these elements matches.
func (this *Selection) IsNodes(nodes ...*html.Node) bool {
return this.FilterNodes(nodes...).Length() > 0
}
// HasClass() determines whether any of the matched elements are assigned the
// given class.
func (this *Selection) HasClass(class string) bool {
class = " " + class + " "
for _, n := range this.Nodes {
// Applies only to element nodes
if n.Type == html.ElementNode {
if elClass, ok := getAttributeValue("class", n); ok {
elClass = rxClassTrim.ReplaceAllString(" "+elClass+" ", " ")
if strings.Index(elClass, class) > -1 {
return true
}
}
}
}
return false
}
// Contains() returns true if the specified Node is within,
// at any depth, one of the nodes in the Selection object.
// It is NOT inclusive, to behave like jQuery's implementation, and
// unlike Javascript's .contains(), so if the contained
// node is itself in the selection, it returns false.
func (this *Selection) Contains(n *html.Node) bool {
return sliceContains(this.Nodes, n)
}
+537
View File
@@ -0,0 +1,537 @@
package goquery
import (
"code.google.com/p/cascadia"
"code.google.com/p/go.net/html"
)
type siblingType int
// Sibling type, used internally when iterating over children at the same
// level (siblings) to specify which nodes are requested.
const (
siblingPrevUntil siblingType = iota - 3
siblingPrevAll
siblingPrev
siblingAll
siblingNext
siblingNextAll
siblingNextUntil
siblingAllIncludingNonElements
)
// Find() gets the descendants of each element in the current set of matched
// elements, filtered by a selector. It returns a new Selection object
// containing these matched elements.
func (this *Selection) Find(selector string) *Selection {
return pushStack(this, findWithSelector(this.Nodes, selector))
}
// FindSelection() gets the descendants of each element in the current
// Selection, filtered by a Selection. It returns a new Selection object
// containing these matched elements.
func (this *Selection) FindSelection(sel *Selection) *Selection {
if sel == nil {
return pushStack(this, nil)
}
return this.FindNodes(sel.Nodes...)
}
// FindNodes() gets the descendants of each element in the current
// Selection, filtered by some nodes. It returns a new Selection object
// containing these matched elements.
func (this *Selection) FindNodes(nodes ...*html.Node) *Selection {
return pushStack(this, mapNodes(nodes, func(i int, n *html.Node) []*html.Node {
if sliceContains(this.Nodes, n) {
return []*html.Node{n}
}
return nil
}))
}
// Contents() gets the children of each element in the Selection,
// including text and comment nodes. It returns a new Selection object
// containing these elements.
func (this *Selection) Contents() *Selection {
return pushStack(this, getChildrenNodes(this.Nodes, siblingAllIncludingNonElements))
}
// ContentsFiltered() gets the children of each element in the Selection,
// filtered by the specified selector. It returns a new Selection
// object containing these elements. Since selectors only act on Element nodes,
// this function is an alias to ChildrenFiltered() unless the selector is empty,
// in which case it is an alias to Contents().
func (this *Selection) ContentsFiltered(selector string) *Selection {
if selector != "" {
return this.ChildrenFiltered(selector)
}
return this.Contents()
}
// Children() gets the child elements of each element in the Selection.
// It returns a new Selection object containing these elements.
func (this *Selection) Children() *Selection {
return pushStack(this, getChildrenNodes(this.Nodes, siblingAll))
}
// ChildrenFiltered() gets the child elements of each element in the Selection,
// filtered by the specified selector. It returns a new
// Selection object containing these elements.
func (this *Selection) ChildrenFiltered(selector string) *Selection {
return filterAndPush(this, getChildrenNodes(this.Nodes, siblingAll), selector)
}
// Parent() gets the parent of each element in the Selection. It returns a
// new Selection object containing the matched elements.
func (this *Selection) Parent() *Selection {
return pushStack(this, getParentNodes(this.Nodes))
}
// ParentFiltered() gets the parent of each element in the Selection filtered by a
// selector. It returns a new Selection object containing the matched elements.
func (this *Selection) ParentFiltered(selector string) *Selection {
return filterAndPush(this, getParentNodes(this.Nodes), selector)
}
// Closest() gets the first element that matches the selector by testing the
// element itself and traversing up through its ancestors in the DOM tree.
func (this *Selection) Closest(selector string) *Selection {
cs := cascadia.MustCompile(selector)
return pushStack(this, mapNodes(this.Nodes, func(i int, n *html.Node) []*html.Node {
// For each node in the selection, test the node itself, then each parent
// until a match is found.
for ; n != nil; n = n.Parent {
if cs.Match(n) {
return []*html.Node{n}
}
}
return nil
}))
}
// ClosestNodes() gets the first element that matches one of the nodes by testing the
// element itself and traversing up through its ancestors in the DOM tree.
func (this *Selection) ClosestNodes(nodes ...*html.Node) *Selection {
return pushStack(this, mapNodes(this.Nodes, func(i int, n *html.Node) []*html.Node {
// For each node in the selection, test the node itself, then each parent
// until a match is found.
for ; n != nil; n = n.Parent {
if isInSlice(nodes, n) {
return []*html.Node{n}
}
}
return nil
}))
}
// ClosestSelection() gets the first element that matches one of the nodes in the
// Selection by testing the element itself and traversing up through its ancestors
// in the DOM tree.
func (this *Selection) ClosestSelection(s *Selection) *Selection {
if s == nil {
return pushStack(this, nil)
}
return this.ClosestNodes(s.Nodes...)
}
// Parents() gets the ancestors of each element in the current Selection. It
// returns a new Selection object with the matched elements.
func (this *Selection) Parents() *Selection {
return pushStack(this, getParentsNodes(this.Nodes, "", nil))
}
// ParentsFiltered() gets the ancestors of each element in the current
// Selection. It returns a new Selection object with the matched elements.
func (this *Selection) ParentsFiltered(selector string) *Selection {
return filterAndPush(this, getParentsNodes(this.Nodes, "", nil), selector)
}
// ParentsUntil() gets the ancestors of each element in the Selection, up to but
// not including the element matched by the selector. It returns a new Selection
// object containing the matched elements.
func (this *Selection) ParentsUntil(selector string) *Selection {
return pushStack(this, getParentsNodes(this.Nodes, selector, nil))
}
// ParentsUntilSelection() gets the ancestors of each element in the Selection,
// up to but not including the elements in the specified Selection. It returns a
// new Selection object containing the matched elements.
func (this *Selection) ParentsUntilSelection(sel *Selection) *Selection {
if sel == nil {
return this.Parents()
}
return this.ParentsUntilNodes(sel.Nodes...)
}
// ParentsUntilNodes() gets the ancestors of each element in the Selection,
// up to but not including the specified nodes. It returns a
// new Selection object containing the matched elements.
func (this *Selection) ParentsUntilNodes(nodes ...*html.Node) *Selection {
return pushStack(this, getParentsNodes(this.Nodes, "", nodes))
}
// ParentsFilteredUntil() is like ParentsUntil(), with the option to filter the
// results based on a selector string. It returns a new Selection
// object containing the matched elements.
func (this *Selection) ParentsFilteredUntil(filterSelector string, untilSelector string) *Selection {
return filterAndPush(this, getParentsNodes(this.Nodes, untilSelector, nil), filterSelector)
}
// ParentsFilteredUntilSelection() is like ParentsUntilSelection(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) ParentsFilteredUntilSelection(filterSelector string, sel *Selection) *Selection {
if sel == nil {
return this.ParentsFiltered(filterSelector)
}
return this.ParentsFilteredUntilNodes(filterSelector, sel.Nodes...)
}
// ParentsFilteredUntilNodes() is like ParentsUntilNodes(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) ParentsFilteredUntilNodes(filterSelector string, nodes ...*html.Node) *Selection {
return filterAndPush(this, getParentsNodes(this.Nodes, "", nodes), filterSelector)
}
// Siblings() gets the siblings of each element in the Selection. It returns
// a new Selection object containing the matched elements.
func (this *Selection) Siblings() *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingAll, "", nil))
}
// SiblingsFiltered() gets the siblings of each element in the Selection
// filtered by a selector. It returns a new Selection object containing the
// matched elements.
func (this *Selection) SiblingsFiltered(selector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingAll, "", nil), selector)
}
// Next() gets the immediately following sibling of each element in the
// Selection. It returns a new Selection object containing the matched elements.
func (this *Selection) Next() *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingNext, "", nil))
}
// NextFiltered() gets the immediately following sibling of each element in the
// Selection filtered by a selector. It returns a new Selection object
// containing the matched elements.
func (this *Selection) NextFiltered(selector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingNext, "", nil), selector)
}
// NextAll() gets all the following siblings of each element in the
// Selection. It returns a new Selection object containing the matched elements.
func (this *Selection) NextAll() *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingNextAll, "", nil))
}
// NextAllFiltered() gets all the following siblings of each element in the
// Selection filtered by a selector. It returns a new Selection object
// containing the matched elements.
func (this *Selection) NextAllFiltered(selector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingNextAll, "", nil), selector)
}
// Prev() gets the immediately preceding sibling of each element in the
// Selection. It returns a new Selection object containing the matched elements.
func (this *Selection) Prev() *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingPrev, "", nil))
}
// PrevFiltered() gets the immediately preceding sibling of each element in the
// Selection filtered by a selector. It returns a new Selection object
// containing the matched elements.
func (this *Selection) PrevFiltered(selector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingPrev, "", nil), selector)
}
// PrevAll() gets all the preceding siblings of each element in the
// Selection. It returns a new Selection object containing the matched elements.
func (this *Selection) PrevAll() *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingPrevAll, "", nil))
}
// PrevAllFiltered() gets all the preceding siblings of each element in the
// Selection filtered by a selector. It returns a new Selection object
// containing the matched elements.
func (this *Selection) PrevAllFiltered(selector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingPrevAll, "", nil), selector)
}
// NextUntil() gets all following siblings of each element up to but not
// including the element matched by the selector. It returns a new Selection
// object containing the matched elements.
func (this *Selection) NextUntil(selector string) *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingNextUntil,
selector, nil))
}
// NextUntilSelection() gets all following siblings of each element up to but not
// including the element matched by the Selection. It returns a new Selection
// object containing the matched elements.
func (this *Selection) NextUntilSelection(sel *Selection) *Selection {
if sel == nil {
return this.NextAll()
}
return this.NextUntilNodes(sel.Nodes...)
}
// NextUntilNodes() gets all following siblings of each element up to but not
// including the element matched by the nodes. It returns a new Selection
// object containing the matched elements.
func (this *Selection) NextUntilNodes(nodes ...*html.Node) *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingNextUntil,
"", nodes))
}
// PrevUntil() gets all preceding siblings of each element up to but not
// including the element matched by the selector. It returns a new Selection
// object containing the matched elements.
func (this *Selection) PrevUntil(selector string) *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingPrevUntil,
selector, nil))
}
// PrevUntilSelection() gets all preceding siblings of each element up to but not
// including the element matched by the Selection. It returns a new Selection
// object containing the matched elements.
func (this *Selection) PrevUntilSelection(sel *Selection) *Selection {
if sel == nil {
return this.PrevAll()
}
return this.PrevUntilNodes(sel.Nodes...)
}
// PrevUntilNodes() gets all preceding siblings of each element up to but not
// including the element matched by the nodes. It returns a new Selection
// object containing the matched elements.
func (this *Selection) PrevUntilNodes(nodes ...*html.Node) *Selection {
return pushStack(this, getSiblingNodes(this.Nodes, siblingPrevUntil,
"", nodes))
}
// NextFilteredUntil() is like NextUntil(), with the option to filter
// the results based on a selector string.
// It returns a new Selection object containing the matched elements.
func (this *Selection) NextFilteredUntil(filterSelector string, untilSelector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingNextUntil,
untilSelector, nil), filterSelector)
}
// NextFilteredUntilSelection() is like NextUntilSelection(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) NextFilteredUntilSelection(filterSelector string, sel *Selection) *Selection {
if sel == nil {
return this.NextFiltered(filterSelector)
}
return this.NextFilteredUntilNodes(filterSelector, sel.Nodes...)
}
// NextFilteredUntilNodes() is like NextUntilNodes(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) NextFilteredUntilNodes(filterSelector string, nodes ...*html.Node) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingNextUntil,
"", nodes), filterSelector)
}
// PrevFilteredUntil() is like PrevUntil(), with the option to filter
// the results based on a selector string.
// It returns a new Selection object containing the matched elements.
func (this *Selection) PrevFilteredUntil(filterSelector string, untilSelector string) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingPrevUntil,
untilSelector, nil), filterSelector)
}
// PrevFilteredUntilSelection() is like PrevUntilSelection(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) PrevFilteredUntilSelection(filterSelector string, sel *Selection) *Selection {
if sel == nil {
return this.PrevFiltered(filterSelector)
}
return this.PrevFilteredUntilNodes(filterSelector, sel.Nodes...)
}
// PrevFilteredUntilNodes() is like PrevUntilNodes(), with the
// option to filter the results based on a selector string. It returns a new
// Selection object containing the matched elements.
func (this *Selection) PrevFilteredUntilNodes(filterSelector string, nodes ...*html.Node) *Selection {
return filterAndPush(this, getSiblingNodes(this.Nodes, siblingPrevUntil,
"", nodes), filterSelector)
}
// Filter and push filters the nodes based on a selector, and pushes the results
// on the stack, with the srcSel as previous selection.
func filterAndPush(srcSel *Selection, nodes []*html.Node, selector string) *Selection {
// Create a temporary Selection with the specified nodes to filter using winnow
sel := &Selection{nodes, srcSel.document, nil}
// Filter based on selector and push on stack
return pushStack(srcSel, winnow(sel, selector, true))
}
// Internal implementation of Find that return raw nodes.
func findWithSelector(nodes []*html.Node, selector string) []*html.Node {
// Compile the selector once
sel := cascadia.MustCompile(selector)
// Map nodes to find the matches within the children of each node
return mapNodes(nodes, func(i int, n *html.Node) (result []*html.Node) {
// Go down one level, becausejQuery's Find() selects only within descendants
for c := n.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.ElementNode {
result = append(result, sel.MatchAll(c)...)
}
}
return
})
}
// Internal implementation to get all parent nodes, stopping at the specified
// node (or nil if no stop).
func getParentsNodes(nodes []*html.Node, stopSelector string, stopNodes []*html.Node) []*html.Node {
return mapNodes(nodes, func(i int, n *html.Node) (result []*html.Node) {
for p := n.Parent; p != nil; p = p.Parent {
sel := newSingleSelection(p, nil)
if stopSelector != "" {
if sel.Is(stopSelector) {
break
}
} else if len(stopNodes) > 0 {
if sel.IsNodes(stopNodes...) {
break
}
}
if p.Type == html.ElementNode {
result = append(result, p)
}
}
return
})
}
// Internal implementation of sibling nodes that return a raw slice of matches.
func getSiblingNodes(nodes []*html.Node, st siblingType, untilSelector string, untilNodes []*html.Node) []*html.Node {
var f func(*html.Node) bool
// If the requested siblings are ...Until(), create the test function to
// determine if the until condition is reached (returns true if it is)
if st == siblingNextUntil || st == siblingPrevUntil {
f = func(n *html.Node) bool {
if untilSelector != "" {
// Selector-based condition
sel := newSingleSelection(n, nil)
return sel.Is(untilSelector)
} else if len(untilNodes) > 0 {
// Nodes-based condition
sel := newSingleSelection(n, nil)
return sel.IsNodes(untilNodes...)
}
return false
}
}
return mapNodes(nodes, func(i int, n *html.Node) []*html.Node {
return getChildrenWithSiblingType(n.Parent, st, n, f)
})
}
// Gets the children nodes of each node in the specified slice of nodes,
// based on the sibling type request.
func getChildrenNodes(nodes []*html.Node, st siblingType) []*html.Node {
return mapNodes(nodes, func(i int, n *html.Node) []*html.Node {
return getChildrenWithSiblingType(n, st, nil, nil)
})
}
// Gets the children of the specified parent, based on the requested sibling
// type, skipping a specified node if required.
func getChildrenWithSiblingType(parent *html.Node, st siblingType, skipNode *html.Node,
untilFunc func(*html.Node) bool) (result []*html.Node) {
// Create the iterator function
var iter = func(cur *html.Node) (ret *html.Node) {
// Based on the sibling type requested, iterate the right way
for {
switch st {
case siblingAll, siblingAllIncludingNonElements:
if cur == nil {
// First iteration, start with first child of parent
// Skip node if required
if ret = parent.FirstChild; ret == skipNode && skipNode != nil {
ret = skipNode.NextSibling
}
} else {
// Skip node if required
if ret = cur.NextSibling; ret == skipNode && skipNode != nil {
ret = skipNode.NextSibling
}
}
case siblingPrev, siblingPrevAll, siblingPrevUntil:
if cur == nil {
// Start with previous sibling of the skip node
ret = skipNode.PrevSibling
} else {
ret = cur.PrevSibling
}
case siblingNext, siblingNextAll, siblingNextUntil:
if cur == nil {
// Start with next sibling of the skip node
ret = skipNode.NextSibling
} else {
ret = cur.NextSibling
}
default:
panic("Invalid sibling type.")
}
if ret == nil || ret.Type == html.ElementNode || st == siblingAllIncludingNonElements {
return
} else {
// Not a valid node, try again from this one
cur = ret
}
}
panic("Unreachable code reached.")
}
for c := iter(nil); c != nil; c = iter(c) {
// If this is an ...Until() case, test before append (returns true
// if the until condition is reached)
if st == siblingNextUntil || st == siblingPrevUntil {
if untilFunc(c) {
return
}
}
result = append(result, c)
if st == siblingNext || st == siblingPrev {
// Only one node was requested (immediate next or previous), so exit
return
}
}
return
}
// Internal implementation of parent nodes that return a raw slice of Nodes.
func getParentNodes(nodes []*html.Node) []*html.Node {
return mapNodes(nodes, func(i int, n *html.Node) []*html.Node {
if n.Parent != nil && n.Parent.Type == html.ElementNode {
return []*html.Node{n.Parent}
}
return nil
})
}
// Internal map function used by many traversing methods. Takes the source nodes
// to iterate on and the mapping function that returns an array of nodes.
// Returns an array of nodes mapped by calling the callback function once for
// each node in the source nodes.
func mapNodes(nodes []*html.Node, f func(int, *html.Node) []*html.Node) (result []*html.Node) {
for i, n := range nodes {
if vals := f(i, n); len(vals) > 0 {
result = appendWithoutDuplicates(result, vals)
}
}
return
}
+93
View File
@@ -0,0 +1,93 @@
package goquery
import (
"code.google.com/p/go.net/html"
"io"
"net/http"
"net/url"
)
// Document represents an HTML document to be manipulated. Unlike jQuery, which
// is loaded as part of a DOM document, and thus acts upon its containing
// document, GoQuery doesn't know which HTML document to act upon. So it needs
// to be told, and that's what the Document class is for. It holds the root
// document node to manipulate, and can make selections on this document.
type Document struct {
*Selection
Url *url.URL
rootNode *html.Node
}
// NewDocumentFromNode() is a Document constructor that takes a root html Node
// as argument.
func NewDocumentFromNode(root *html.Node) (d *Document) {
return newDocument(root, nil)
}
// NewDocument() is a Document constructor that takes a string URL as argument.
// It loads the specified document, parses it, and stores the root Document
// node, ready to be manipulated.
func NewDocument(url string) (d *Document, e error) {
// Load the URL
res, e := http.Get(url)
if e != nil {
return
}
return NewDocumentFromResponse(res)
}
// NewDocumentFromReader() returns a Document from a generic reader.
// It returns an error as second value if the reader's data cannot be parsed
// as html. It does *not* check if the reader is also an io.Closer, so the
// provided reader is never closed by this call, it is the responsibility
// of the caller to close it if required.
func NewDocumentFromReader(r io.Reader) (d *Document, e error) {
root, e := html.Parse(r)
if e != nil {
return nil, e
}
return newDocument(root, nil), nil
}
// NewDocumentFromResponse() is another Document constructor that takes an http resonse as argument.
// It loads the specified response's document, parses it, and stores the root Document
// node, ready to be manipulated.
func NewDocumentFromResponse(res *http.Response) (d *Document, e error) {
defer res.Body.Close()
// Parse the HTML into nodes
root, e := html.Parse(res.Body)
if e != nil {
return
}
// Create and fill the document
return newDocument(root, res.Request.URL), nil
}
// Private constructor, make sure all fields are correctly filled.
func newDocument(root *html.Node, url *url.URL) *Document {
// Create and fill the document
d := &Document{nil, url, root}
d.Selection = newSingleSelection(root, d)
return d
}
// Selection represents a collection of nodes matching some criteria. The
// initial Selection can be created by using Document.Find(), and then
// manipulated using the jQuery-like chainable syntax and methods.
type Selection struct {
Nodes []*html.Node
document *Document
prevSel *Selection
}
// Helper constructor to create an empty selection
func newEmptySelection(doc *Document) *Selection {
return &Selection{nil, doc, nil}
}
// Helper constructor to create a selection of only one node
func newSingleSelection(node *html.Node, doc *Document) *Selection {
return &Selection{[]*html.Node{node}, doc, nil}
}
+87
View File
@@ -0,0 +1,87 @@
package goquery
import (
"code.google.com/p/go.net/html"
)
func getChildren(n *html.Node) (result []*html.Node) {
for c := n.FirstChild; c != nil; c = c.NextSibling {
result = append(result, c)
}
return
}
// Loop through all container nodes to search for the target node.
func sliceContains(container []*html.Node, contained *html.Node) bool {
for _, n := range container {
if nodeContains(n, contained) {
return true
}
}
return false
}
// Checks if the contained node is within the container node.
func nodeContains(container *html.Node, contained *html.Node) bool {
// Check if the parent of the contained node is the container node, traversing
// upward until the top is reached, or the container is found.
for contained = contained.Parent; contained != nil; contained = contained.Parent {
if container == contained {
return true
}
}
return false
}
// Checks if the target node is in the slice of nodes.
func isInSlice(slice []*html.Node, node *html.Node) bool {
return indexInSlice(slice, node) > -1
}
// Returns the index of the target node in the slice, or -1.
func indexInSlice(slice []*html.Node, node *html.Node) int {
if node != nil {
for i, n := range slice {
if n == node {
return i
}
}
}
return -1
}
// Appends the new nodes to the target slice, making sure no duplicate is added.
// There is no check to the original state of the target slice, so it may still
// contain duplicates. The target slice is returned because append() may create
// a new underlying array.
func appendWithoutDuplicates(target []*html.Node,
nodes []*html.Node) []*html.Node {
for _, n := range nodes {
if !isInSlice(target, n) {
target = append(target, n)
}
}
return target
}
// Loop through a selection, returning only those nodes that pass the predicate
// function.
func grep(sel *Selection, predicate func(i int, s *Selection) bool) (result []*html.Node) {
for i, n := range sel.Nodes {
if predicate(i, newSingleSelection(n, sel.document)) {
result = append(result, n)
}
}
return
}
// Creates a new Selection object based on the specified nodes, and keeps the
// source Selection object on the stack (linked list).
func pushStack(fromSel *Selection, nodes []*html.Node) (result *Selection) {
result = &Selection{nodes, fromSel.document, fromSel}
return
}
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Chris Hines
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+38
View File
@@ -0,0 +1,38 @@
[![GoDoc](https://godoc.org/github.com/go-stack/stack?status.svg)](https://godoc.org/github.com/go-stack/stack)
[![Go Report Card](https://goreportcard.com/badge/go-stack/stack)](https://goreportcard.com/report/go-stack/stack)
[![TravisCI](https://travis-ci.org/go-stack/stack.svg?branch=master)](https://travis-ci.org/go-stack/stack)
[![Coverage Status](https://coveralls.io/repos/github/go-stack/stack/badge.svg?branch=master)](https://coveralls.io/github/go-stack/stack?branch=master)
# stack
Package stack implements utilities to capture, manipulate, and format call
stacks. It provides a simpler API than package runtime.
The implementation takes care of the minutia and special cases of interpreting
the program counter (pc) values returned by runtime.Callers.
## Versioning
Package stack publishes releases via [semver](http://semver.org/) compatible Git
tags prefixed with a single 'v'. The master branch always contains the latest
release. The develop branch contains unreleased commits.
## Formatting
Package stack's types implement fmt.Formatter, which provides a simple and
flexible way to declaratively configure formatting when used with logging or
error tracking packages.
```go
func DoTheThing() {
c := stack.Caller(0)
log.Print(c) // "source.go:10"
log.Printf("%+v", c) // "pkg/path/source.go:10"
log.Printf("%n", c) // "DoTheThing"
s := stack.Trace().TrimRuntime()
log.Print(s) // "[source.go:15 caller.go:42 main.go:14]"
}
```
See the docs for all of the supported formatting options.
+324
View File
@@ -0,0 +1,324 @@
// +build go1.7
// Package stack implements utilities to capture, manipulate, and format call
// stacks. It provides a simpler API than package runtime.
//
// The implementation takes care of the minutia and special cases of
// interpreting the program counter (pc) values returned by runtime.Callers.
//
// Package stack's types implement fmt.Formatter, which provides a simple and
// flexible way to declaratively configure formatting when used with logging
// or error tracking packages.
package stack
import (
"bytes"
"errors"
"fmt"
"io"
"runtime"
"strconv"
"strings"
)
// Call records a single function invocation from a goroutine stack.
type Call struct {
frame runtime.Frame
}
// Caller returns a Call from the stack of the current goroutine. The argument
// skip is the number of stack frames to ascend, with 0 identifying the
// calling function.
func Caller(skip int) Call {
// As of Go 1.9 we need room for up to three PC entries.
//
// 0. An entry for the stack frame prior to the target to check for
// special handling needed if that prior entry is runtime.sigpanic.
// 1. A possible second entry to hold metadata about skipped inlined
// functions. If inline functions were not skipped the target frame
// PC will be here.
// 2. A third entry for the target frame PC when the second entry
// is used for skipped inline functions.
var pcs [3]uintptr
n := runtime.Callers(skip+1, pcs[:])
frames := runtime.CallersFrames(pcs[:n])
frame, _ := frames.Next()
frame, _ = frames.Next()
return Call{
frame: frame,
}
}
// String implements fmt.Stinger. It is equivalent to fmt.Sprintf("%v", c).
func (c Call) String() string {
return fmt.Sprint(c)
}
// MarshalText implements encoding.TextMarshaler. It formats the Call the same
// as fmt.Sprintf("%v", c).
func (c Call) MarshalText() ([]byte, error) {
if c.frame == (runtime.Frame{}) {
return nil, ErrNoFunc
}
buf := bytes.Buffer{}
fmt.Fprint(&buf, c)
return buf.Bytes(), nil
}
// ErrNoFunc means that the Call has a nil *runtime.Func. The most likely
// cause is a Call with the zero value.
var ErrNoFunc = errors.New("no call stack information")
// Format implements fmt.Formatter with support for the following verbs.
//
// %s source file
// %d line number
// %n function name
// %k last segment of the package path
// %v equivalent to %s:%d
//
// It accepts the '+' and '#' flags for most of the verbs as follows.
//
// %+s path of source file relative to the compile time GOPATH
// %#s full path of source file
// %+n import path qualified function name
// %+k full package path
// %+v equivalent to %+s:%d
// %#v equivalent to %#s:%d
func (c Call) Format(s fmt.State, verb rune) {
if c.frame == (runtime.Frame{}) {
fmt.Fprintf(s, "%%!%c(NOFUNC)", verb)
return
}
switch verb {
case 's', 'v':
file := c.frame.File
switch {
case s.Flag('#'):
// done
case s.Flag('+'):
file = file[pkgIndex(file, c.frame.Function):]
default:
const sep = "/"
if i := strings.LastIndex(file, sep); i != -1 {
file = file[i+len(sep):]
}
}
io.WriteString(s, file)
if verb == 'v' {
buf := [7]byte{':'}
s.Write(strconv.AppendInt(buf[:1], int64(c.frame.Line), 10))
}
case 'd':
buf := [6]byte{}
s.Write(strconv.AppendInt(buf[:0], int64(c.frame.Line), 10))
case 'k':
name := c.frame.Function
const pathSep = "/"
start, end := 0, len(name)
if i := strings.LastIndex(name, pathSep); i != -1 {
start = i + len(pathSep)
}
const pkgSep = "."
if i := strings.Index(name[start:], pkgSep); i != -1 {
end = start + i
}
if s.Flag('+') {
start = 0
}
io.WriteString(s, name[start:end])
case 'n':
name := c.frame.Function
if !s.Flag('+') {
const pathSep = "/"
if i := strings.LastIndex(name, pathSep); i != -1 {
name = name[i+len(pathSep):]
}
const pkgSep = "."
if i := strings.Index(name, pkgSep); i != -1 {
name = name[i+len(pkgSep):]
}
}
io.WriteString(s, name)
}
}
// Frame returns the call frame infomation for the Call.
func (c Call) Frame() runtime.Frame {
return c.frame
}
// PC returns the program counter for this call frame; multiple frames may
// have the same PC value.
//
// Deprecated: Use Call.Frame instead.
func (c Call) PC() uintptr {
return c.frame.PC
}
// CallStack records a sequence of function invocations from a goroutine
// stack.
type CallStack []Call
// String implements fmt.Stinger. It is equivalent to fmt.Sprintf("%v", cs).
func (cs CallStack) String() string {
return fmt.Sprint(cs)
}
var (
openBracketBytes = []byte("[")
closeBracketBytes = []byte("]")
spaceBytes = []byte(" ")
)
// MarshalText implements encoding.TextMarshaler. It formats the CallStack the
// same as fmt.Sprintf("%v", cs).
func (cs CallStack) MarshalText() ([]byte, error) {
buf := bytes.Buffer{}
buf.Write(openBracketBytes)
for i, pc := range cs {
if i > 0 {
buf.Write(spaceBytes)
}
fmt.Fprint(&buf, pc)
}
buf.Write(closeBracketBytes)
return buf.Bytes(), nil
}
// Format implements fmt.Formatter by printing the CallStack as square brackets
// ([, ]) surrounding a space separated list of Calls each formatted with the
// supplied verb and options.
func (cs CallStack) Format(s fmt.State, verb rune) {
s.Write(openBracketBytes)
for i, pc := range cs {
if i > 0 {
s.Write(spaceBytes)
}
pc.Format(s, verb)
}
s.Write(closeBracketBytes)
}
// Trace returns a CallStack for the current goroutine with element 0
// identifying the calling function.
func Trace() CallStack {
var pcs [512]uintptr
n := runtime.Callers(1, pcs[:])
frames := runtime.CallersFrames(pcs[:n])
cs := make(CallStack, 0, n)
// Skip extra frame retrieved just to make sure the runtime.sigpanic
// special case is handled.
frame, more := frames.Next()
for more {
frame, more = frames.Next()
cs = append(cs, Call{frame: frame})
}
return cs
}
// TrimBelow returns a slice of the CallStack with all entries below c
// removed.
func (cs CallStack) TrimBelow(c Call) CallStack {
for len(cs) > 0 && cs[0] != c {
cs = cs[1:]
}
return cs
}
// TrimAbove returns a slice of the CallStack with all entries above c
// removed.
func (cs CallStack) TrimAbove(c Call) CallStack {
for len(cs) > 0 && cs[len(cs)-1] != c {
cs = cs[:len(cs)-1]
}
return cs
}
// pkgIndex returns the index that results in file[index:] being the path of
// file relative to the compile time GOPATH, and file[:index] being the
// $GOPATH/src/ portion of file. funcName must be the name of a function in
// file as returned by runtime.Func.Name.
func pkgIndex(file, funcName string) int {
// As of Go 1.6.2 there is no direct way to know the compile time GOPATH
// at runtime, but we can infer the number of path segments in the GOPATH.
// We note that runtime.Func.Name() returns the function name qualified by
// the import path, which does not include the GOPATH. Thus we can trim
// segments from the beginning of the file path until the number of path
// separators remaining is one more than the number of path separators in
// the function name. For example, given:
//
// GOPATH /home/user
// file /home/user/src/pkg/sub/file.go
// fn.Name() pkg/sub.Type.Method
//
// We want to produce:
//
// file[:idx] == /home/user/src/
// file[idx:] == pkg/sub/file.go
//
// From this we can easily see that fn.Name() has one less path separator
// than our desired result for file[idx:]. We count separators from the
// end of the file path until it finds two more than in the function name
// and then move one character forward to preserve the initial path
// segment without a leading separator.
const sep = "/"
i := len(file)
for n := strings.Count(funcName, sep) + 2; n > 0; n-- {
i = strings.LastIndex(file[:i], sep)
if i == -1 {
i = -len(sep)
break
}
}
// get back to 0 or trim the leading separator
return i + len(sep)
}
var runtimePath string
func init() {
var pcs [3]uintptr
runtime.Callers(0, pcs[:])
frames := runtime.CallersFrames(pcs[:])
frame, _ := frames.Next()
file := frame.File
idx := pkgIndex(frame.File, frame.Function)
runtimePath = file[:idx]
if runtime.GOOS == "windows" {
runtimePath = strings.ToLower(runtimePath)
}
}
func inGoroot(c Call) bool {
file := c.frame.File
if len(file) == 0 || file[0] == '?' {
return true
}
if runtime.GOOS == "windows" {
file = strings.ToLower(file)
}
return strings.HasPrefix(file, runtimePath) || strings.HasSuffix(file, "/_testmain.go")
}
// TrimRuntime returns a slice of the CallStack with the topmost entries from
// the go runtime removed. It considers any calls originating from unknown
// files, files under GOROOT, or _testmain.go as part of the runtime.
func (cs CallStack) TrimRuntime() CallStack {
for len(cs) > 0 && inGoroot(cs[len(cs)-1]) {
cs = cs[:len(cs)-1]
}
return cs
}
+13
View File
@@ -0,0 +1,13 @@
Copyright 2014 Alan Shreve
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Simon Eskildsen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+5
View File
@@ -0,0 +1,5 @@
// Package term provides utilities for working with terminals.
//
// Deprecated: This package is no longer used by log15. Please use
// github.com/mattn/go-isatty instead.
package term
+13
View File
@@ -0,0 +1,13 @@
// Based on ssh/terminal:
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build appengine
package term
// IsTty always returns false on AppEngine.
func IsTty(fd uintptr) bool {
return false
}
+15
View File
@@ -0,0 +1,15 @@
// Based on ssh/terminal:
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !appengine
package term
import "syscall"
const ioctlReadTermios = syscall.TIOCGETA
// Termios functions describe a general terminal interface that is
// provided to control asynchronous communications ports.
type Termios syscall.Termios
+20
View File
@@ -0,0 +1,20 @@
package term
import (
"syscall"
)
const ioctlReadTermios = syscall.TIOCGETA
// Termios functions describe a general terminal interface that is
// provided to control asynchronous communications ports.
// Go 1.2 doesn't include Termios for FreeBSD. This should be added in 1.3 and this could be merged with terminal_darwin.
type Termios struct {
Iflag uint32
Oflag uint32
Cflag uint32
Lflag uint32
Cc [20]uint8
Ispeed uint32
Ospeed uint32
}
+16
View File
@@ -0,0 +1,16 @@
// Based on ssh/terminal:
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !appengine
package term
import "syscall"
const ioctlReadTermios = syscall.TCGETS
// Termios functions describe a general terminal interface that is
// provided to control asynchronous communications ports.
type Termios syscall.Termios
+9
View File
@@ -0,0 +1,9 @@
package term
import "syscall"
const ioctlReadTermios = syscall.TIOCGETA
// Termios functions describe a general terminal interface that is
// provided to control asynchronous communications ports.
type Termios syscall.Termios
+20
View File
@@ -0,0 +1,20 @@
// Based on ssh/terminal:
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build linux,!appengine darwin freebsd openbsd netbsd
package term
import (
"syscall"
"unsafe"
)
// IsTty returns true if the given file descriptor is a terminal.
func IsTty(fd uintptr) bool {
var termios Termios
_, _, err := syscall.Syscall6(syscall.SYS_IOCTL, fd, ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0)
return err == 0
}
+9
View File
@@ -0,0 +1,9 @@
package term
import "syscall"
const ioctlReadTermios = syscall.TIOCGETA
// Termios functions describe a general terminal interface that is
// provided to control asynchronous communications ports.
type Termios syscall.Termios
+9
View File
@@ -0,0 +1,9 @@
package term
import "golang.org/x/sys/unix"
// IsTty returns true if the given file descriptor is a terminal.
func IsTty(fd uintptr) bool {
_, err := unix.IoctlGetTermios(int(fd), unix.TCGETA)
return err == nil
}
+26
View File
@@ -0,0 +1,26 @@
// Based on ssh/terminal:
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build windows
package term
import (
"syscall"
"unsafe"
)
var kernel32 = syscall.NewLazyDLL("kernel32.dll")
var (
procGetConsoleMode = kernel32.NewProc("GetConsoleMode")
)
// IsTty returns true if the given file descriptor is a terminal.
func IsTty(fd uintptr) bool {
var st uint32
r, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, fd, uintptr(unsafe.Pointer(&st)), 0)
return r != 0 && e == 0
}
+27
View File
@@ -0,0 +1,27 @@
Copyright (c) 2012 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+32
View File
@@ -0,0 +1,32 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// forwardCopy is like the built-in copy function except that it always goes
// forward from the start, even if the dst and src overlap.
// It is equivalent to:
// for i := 0; i < n; i++ {
// mem[dst+i] = mem[src+i]
// }
func forwardCopy(mem []byte, dst, src, n int) {
if dst <= src {
copy(mem[dst:dst+n], mem[src:src+n])
return
}
for {
if dst >= src+n {
copy(mem[dst:dst+n], mem[src:src+n])
return
}
// There is some forward overlap. The destination
// will be filled with a repeated pattern of mem[src:src+k].
// We copy one instance of the pattern here, then repeat.
// Each time around this loop k will double.
k := dst - src
copy(mem[dst:dst+k], mem[src:src+k])
n -= k
dst += k
}
}
+41
View File
@@ -0,0 +1,41 @@
//+build !noasm
//+build !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package flate
import (
"github.com/klauspost/cpuid"
)
// crc32sse returns a hash for the first 4 bytes of the slice
// len(a) must be >= 4.
//go:noescape
func crc32sse(a []byte) uint32
// crc32sseAll calculates hashes for each 4-byte set in a.
// dst must be east len(a) - 4 in size.
// The size is not checked by the assembly.
//go:noescape
func crc32sseAll(a []byte, dst []uint32)
// matchLenSSE4 returns the number of matching bytes in a and b
// up to length 'max'. Both slices must be at least 'max'
// bytes in size.
//
// TODO: drop the "SSE4" name, since it doesn't use any SSE instructions.
//
//go:noescape
func matchLenSSE4(a, b []byte, max int) int
// histogram accumulates a histogram of b in h.
// h must be at least 256 entries in length,
// and must be cleared before calling this function.
//go:noescape
func histogram(b []byte, h []int32)
// Detect SSE 4.2 feature.
func init() {
useSSE42 = cpuid.CPU.SSE42()
}
+213
View File
@@ -0,0 +1,213 @@
//+build !noasm
//+build !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
// func crc32sse(a []byte) uint32
TEXT ·crc32sse(SB), 4, $0
MOVQ a+0(FP), R10
XORQ BX, BX
// CRC32 dword (R10), EBX
BYTE $0xF2; BYTE $0x41; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0x1a
MOVL BX, ret+24(FP)
RET
// func crc32sseAll(a []byte, dst []uint32)
TEXT ·crc32sseAll(SB), 4, $0
MOVQ a+0(FP), R8 // R8: src
MOVQ a_len+8(FP), R10 // input length
MOVQ dst+24(FP), R9 // R9: dst
SUBQ $4, R10
JS end
JZ one_crc
MOVQ R10, R13
SHRQ $2, R10 // len/4
ANDQ $3, R13 // len&3
XORQ BX, BX
ADDQ $1, R13
TESTQ R10, R10
JZ rem_loop
crc_loop:
MOVQ (R8), R11
XORQ BX, BX
XORQ DX, DX
XORQ DI, DI
MOVQ R11, R12
SHRQ $8, R11
MOVQ R12, AX
MOVQ R11, CX
SHRQ $16, R12
SHRQ $16, R11
MOVQ R12, SI
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
// CRC32 ECX, EDX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd1
// CRC32 ESI, EDI
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xfe
MOVL BX, (R9)
MOVL DX, 4(R9)
MOVL DI, 8(R9)
XORQ BX, BX
MOVL R11, AX
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
MOVL BX, 12(R9)
ADDQ $16, R9
ADDQ $4, R8
XORQ BX, BX
SUBQ $1, R10
JNZ crc_loop
rem_loop:
MOVL (R8), AX
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
MOVL BX, (R9)
ADDQ $4, R9
ADDQ $1, R8
XORQ BX, BX
SUBQ $1, R13
JNZ rem_loop
end:
RET
one_crc:
MOVQ $1, R13
XORQ BX, BX
JMP rem_loop
// func matchLenSSE4(a, b []byte, max int) int
TEXT ·matchLenSSE4(SB), 4, $0
MOVQ a_base+0(FP), SI
MOVQ b_base+24(FP), DI
MOVQ DI, DX
MOVQ max+48(FP), CX
cmp8:
// As long as we are 8 or more bytes before the end of max, we can load and
// compare 8 bytes at a time. If those 8 bytes are equal, repeat.
CMPQ CX, $8
JLT cmp1
MOVQ (SI), AX
MOVQ (DI), BX
CMPQ AX, BX
JNE bsf
ADDQ $8, SI
ADDQ $8, DI
SUBQ $8, CX
JMP cmp8
bsf:
// If those 8 bytes were not equal, XOR the two 8 byte values, and return
// the index of the first byte that differs. The BSF instruction finds the
// least significant 1 bit, the amd64 architecture is little-endian, and
// the shift by 3 converts a bit index to a byte index.
XORQ AX, BX
BSFQ BX, BX
SHRQ $3, BX
ADDQ BX, DI
// Subtract off &b[0] to convert from &b[ret] to ret, and return.
SUBQ DX, DI
MOVQ DI, ret+56(FP)
RET
cmp1:
// In the slices' tail, compare 1 byte at a time.
CMPQ CX, $0
JEQ matchLenEnd
MOVB (SI), AX
MOVB (DI), BX
CMPB AX, BX
JNE matchLenEnd
ADDQ $1, SI
ADDQ $1, DI
SUBQ $1, CX
JMP cmp1
matchLenEnd:
// Subtract off &b[0] to convert from &b[ret] to ret, and return.
SUBQ DX, DI
MOVQ DI, ret+56(FP)
RET
// func histogram(b []byte, h []int32)
TEXT ·histogram(SB), 4, $0
MOVQ b+0(FP), SI // SI: &b
MOVQ b_len+8(FP), R9 // R9: len(b)
MOVQ h+24(FP), DI // DI: Histogram
MOVQ R9, R8
SHRQ $3, R8
JZ hist1
XORQ R11, R11
loop_hist8:
MOVQ (SI), R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
INCL (DI)(R10*4)
ADDQ $8, SI
DECQ R8
JNZ loop_hist8
hist1:
ANDQ $7, R9
JZ end_hist
XORQ R10, R10
loop_hist1:
MOVB (SI), R10
INCL (DI)(R10*4)
INCQ SI
DECQ R9
JNZ loop_hist1
end_hist:
RET
+35
View File
@@ -0,0 +1,35 @@
//+build !amd64 noasm appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package flate
func init() {
useSSE42 = false
}
// crc32sse should never be called.
func crc32sse(a []byte) uint32 {
panic("no assembler")
}
// crc32sseAll should never be called.
func crc32sseAll(a []byte, dst []uint32) {
panic("no assembler")
}
// matchLenSSE4 should never be called.
func matchLenSSE4(a, b []byte, max int) int {
panic("no assembler")
return 0
}
// histogram accumulates a histogram of b in h.
//
// len(h) must be >= 256, and h's elements must be all zeroes.
func histogram(b []byte, h []int32) {
h = h[:256]
for _, t := range b {
h[t]++
}
}
File diff suppressed because it is too large. Load diff
+184
View File
@@ -0,0 +1,184 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// dictDecoder implements the LZ77 sliding dictionary as used in decompression.
// LZ77 decompresses data through sequences of two forms of commands:
//
// * Literal insertions: Runs of one or more symbols are inserted into the data
// stream as is. This is accomplished through the writeByte method for a
// single symbol, or combinations of writeSlice/writeMark for multiple symbols.
// Any valid stream must start with a literal insertion if no preset dictionary
// is used.
//
// * Backward copies: Runs of one or more symbols are copied from previously
// emitted data. Backward copies come as the tuple (dist, length) where dist
// determines how far back in the stream to copy from and length determines how
// many bytes to copy. Note that it is valid for the length to be greater than
// the distance. Since LZ77 uses forward copies, that situation is used to
// perform a form of run-length encoding on repeated runs of symbols.
// The writeCopy and tryWriteCopy are used to implement this command.
//
// For performance reasons, this implementation performs little to no sanity
// checks about the arguments. As such, the invariants documented for each
// method call must be respected.
type dictDecoder struct {
hist []byte // Sliding window history
// Invariant: 0 <= rdPos <= wrPos <= len(hist)
wrPos int // Current output position in buffer
rdPos int // Have emitted hist[:rdPos] already
full bool // Has a full window length been written yet?
}
// init initializes dictDecoder to have a sliding window dictionary of the given
// size. If a preset dict is provided, it will initialize the dictionary with
// the contents of dict.
func (dd *dictDecoder) init(size int, dict []byte) {
*dd = dictDecoder{hist: dd.hist}
if cap(dd.hist) < size {
dd.hist = make([]byte, size)
}
dd.hist = dd.hist[:size]
if len(dict) > len(dd.hist) {
dict = dict[len(dict)-len(dd.hist):]
}
dd.wrPos = copy(dd.hist, dict)
if dd.wrPos == len(dd.hist) {
dd.wrPos = 0
dd.full = true
}
dd.rdPos = dd.wrPos
}
// histSize reports the total amount of historical data in the dictionary.
func (dd *dictDecoder) histSize() int {
if dd.full {
return len(dd.hist)
}
return dd.wrPos
}
// availRead reports the number of bytes that can be flushed by readFlush.
func (dd *dictDecoder) availRead() int {
return dd.wrPos - dd.rdPos
}
// availWrite reports the available amount of output buffer space.
func (dd *dictDecoder) availWrite() int {
return len(dd.hist) - dd.wrPos
}
// writeSlice returns a slice of the available buffer to write data to.
//
// This invariant will be kept: len(s) <= availWrite()
func (dd *dictDecoder) writeSlice() []byte {
return dd.hist[dd.wrPos:]
}
// writeMark advances the writer pointer by cnt.
//
// This invariant must be kept: 0 <= cnt <= availWrite()
func (dd *dictDecoder) writeMark(cnt int) {
dd.wrPos += cnt
}
// writeByte writes a single byte to the dictionary.
//
// This invariant must be kept: 0 < availWrite()
func (dd *dictDecoder) writeByte(c byte) {
dd.hist[dd.wrPos] = c
dd.wrPos++
}
// writeCopy copies a string at a given (dist, length) to the output.
// This returns the number of bytes copied and may be less than the requested
// length if the available space in the output buffer is too small.
//
// This invariant must be kept: 0 < dist <= histSize()
func (dd *dictDecoder) writeCopy(dist, length int) int {
dstBase := dd.wrPos
dstPos := dstBase
srcPos := dstPos - dist
endPos := dstPos + length
if endPos > len(dd.hist) {
endPos = len(dd.hist)
}
// Copy non-overlapping section after destination position.
//
// This section is non-overlapping in that the copy length for this section
// is always less than or equal to the backwards distance. This can occur
// if a distance refers to data that wraps-around in the buffer.
// Thus, a backwards copy is performed here; that is, the exact bytes in
// the source prior to the copy is placed in the destination.
if srcPos < 0 {
srcPos += len(dd.hist)
dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:])
srcPos = 0
}
// Copy possibly overlapping section before destination position.
//
// This section can overlap if the copy length for this section is larger
// than the backwards distance. This is allowed by LZ77 so that repeated
// strings can be succinctly represented using (dist, length) pairs.
// Thus, a forwards copy is performed here; that is, the bytes copied is
// possibly dependent on the resulting bytes in the destination as the copy
// progresses along. This is functionally equivalent to the following:
//
// for i := 0; i < endPos-dstPos; i++ {
// dd.hist[dstPos+i] = dd.hist[srcPos+i]
// }
// dstPos = endPos
//
for dstPos < endPos {
dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:dstPos])
}
dd.wrPos = dstPos
return dstPos - dstBase
}
// tryWriteCopy tries to copy a string at a given (distance, length) to the
// output. This specialized version is optimized for short distances.
//
// This method is designed to be inlined for performance reasons.
//
// This invariant must be kept: 0 < dist <= histSize()
func (dd *dictDecoder) tryWriteCopy(dist, length int) int {
dstPos := dd.wrPos
endPos := dstPos + length
if dstPos < dist || endPos > len(dd.hist) {
return 0
}
dstBase := dstPos
srcPos := dstPos - dist
// Copy possibly overlapping section before destination position.
loop:
dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:dstPos])
if dstPos < endPos {
goto loop // Avoid for-loop so that this function can be inlined
}
dd.wrPos = dstPos
return dstPos - dstBase
}
// readFlush returns a slice of the historical buffer that is ready to be
// emitted to the user. The data returned by readFlush must be fully consumed
// before calling any other dictDecoder methods.
func (dd *dictDecoder) readFlush() []byte {
toRead := dd.hist[dd.rdPos:dd.wrPos]
dd.rdPos = dd.wrPos
if dd.wrPos == len(dd.hist) {
dd.wrPos, dd.rdPos = 0, 0
dd.full = true
}
return toRead
}
+265
View File
@@ -0,0 +1,265 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build ignore
// This program generates fixedhuff.go
// Invoke as
//
// go run gen.go -output fixedhuff.go
package main
import (
"bytes"
"flag"
"fmt"
"go/format"
"io/ioutil"
"log"
)
var filename = flag.String("output", "fixedhuff.go", "output file name")
const maxCodeLen = 16
// Note: the definition of the huffmanDecoder struct is copied from
// inflate.go, as it is private to the implementation.
// chunk & 15 is number of bits
// chunk >> 4 is value, including table link
const (
huffmanChunkBits = 9
huffmanNumChunks = 1 << huffmanChunkBits
huffmanCountMask = 15
huffmanValueShift = 4
)
type huffmanDecoder struct {
min int // the minimum code length
chunks [huffmanNumChunks]uint32 // chunks as described above
links [][]uint32 // overflow links
linkMask uint32 // mask the width of the link table
}
// Initialize Huffman decoding tables from array of code lengths.
// Following this function, h is guaranteed to be initialized into a complete
// tree (i.e., neither over-subscribed nor under-subscribed). The exception is a
// degenerate case where the tree has only a single symbol with length 1. Empty
// trees are permitted.
func (h *huffmanDecoder) init(bits []int) bool {
// Sanity enables additional runtime tests during Huffman
// table construction. It's intended to be used during
// development to supplement the currently ad-hoc unit tests.
const sanity = false
if h.min != 0 {
*h = huffmanDecoder{}
}
// Count number of codes of each length,
// compute min and max length.
var count [maxCodeLen]int
var min, max int
for _, n := range bits {
if n == 0 {
continue
}
if min == 0 || n < min {
min = n
}
if n > max {
max = n
}
count[n]++
}
// Empty tree. The decompressor.huffSym function will fail later if the tree
// is used. Technically, an empty tree is only valid for the HDIST tree and
// not the HCLEN and HLIT tree. However, a stream with an empty HCLEN tree
// is guaranteed to fail since it will attempt to use the tree to decode the
// codes for the HLIT and HDIST trees. Similarly, an empty HLIT tree is
// guaranteed to fail later since the compressed data section must be
// composed of at least one symbol (the end-of-block marker).
if max == 0 {
return true
}
code := 0
var nextcode [maxCodeLen]int
for i := min; i <= max; i++ {
code <<= 1
nextcode[i] = code
code += count[i]
}
// Check that the coding is complete (i.e., that we've
// assigned all 2-to-the-max possible bit sequences).
// Exception: To be compatible with zlib, we also need to
// accept degenerate single-code codings. See also
// TestDegenerateHuffmanCoding.
if code != 1<<uint(max) && !(code == 1 && max == 1) {
return false
}
h.min = min
if max > huffmanChunkBits {
numLinks := 1 << (uint(max) - huffmanChunkBits)
h.linkMask = uint32(numLinks - 1)
// create link tables
link := nextcode[huffmanChunkBits+1] >> 1
h.links = make([][]uint32, huffmanNumChunks-link)
for j := uint(link); j < huffmanNumChunks; j++ {
reverse := int(reverseByte[j>>8]) | int(reverseByte[j&0xff])<<8
reverse >>= uint(16 - huffmanChunkBits)
off := j - uint(link)
if sanity && h.chunks[reverse] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[reverse] = uint32(off<<huffmanValueShift | (huffmanChunkBits + 1))
h.links[off] = make([]uint32, numLinks)
}
}
for i, n := range bits {
if n == 0 {
continue
}
code := nextcode[n]
nextcode[n]++
chunk := uint32(i<<huffmanValueShift | n)
reverse := int(reverseByte[code>>8]) | int(reverseByte[code&0xff])<<8
reverse >>= uint(16 - n)
if n <= huffmanChunkBits {
for off := reverse; off < len(h.chunks); off += 1 << uint(n) {
// We should never need to overwrite
// an existing chunk. Also, 0 is
// never a valid chunk, because the
// lower 4 "count" bits should be
// between 1 and 15.
if sanity && h.chunks[off] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[off] = chunk
}
} else {
j := reverse & (huffmanNumChunks - 1)
if sanity && h.chunks[j]&huffmanCountMask != huffmanChunkBits+1 {
// Longer codes should have been
// associated with a link table above.
panic("impossible: not an indirect chunk")
}
value := h.chunks[j] >> huffmanValueShift
linktab := h.links[value]
reverse >>= huffmanChunkBits
for off := reverse; off < len(linktab); off += 1 << uint(n-huffmanChunkBits) {
if sanity && linktab[off] != 0 {
panic("impossible: overwriting existing chunk")
}
linktab[off] = chunk
}
}
}
if sanity {
// Above we've sanity checked that we never overwrote
// an existing entry. Here we additionally check that
// we filled the tables completely.
for i, chunk := range h.chunks {
if chunk == 0 {
// As an exception, in the degenerate
// single-code case, we allow odd
// chunks to be missing.
if code == 1 && i%2 == 1 {
continue
}
panic("impossible: missing chunk")
}
}
for _, linktab := range h.links {
for _, chunk := range linktab {
if chunk == 0 {
panic("impossible: missing chunk")
}
}
}
}
return true
}
func main() {
flag.Parse()
var h huffmanDecoder
var bits [288]int
initReverseByte()
for i := 0; i < 144; i++ {
bits[i] = 8
}
for i := 144; i < 256; i++ {
bits[i] = 9
}
for i := 256; i < 280; i++ {
bits[i] = 7
}
for i := 280; i < 288; i++ {
bits[i] = 8
}
h.init(bits[:])
if h.links != nil {
log.Fatal("Unexpected links table in fixed Huffman decoder")
}
var buf bytes.Buffer
fmt.Fprintf(&buf, `// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.`+"\n\n")
fmt.Fprintln(&buf, "package flate")
fmt.Fprintln(&buf)
fmt.Fprintln(&buf, "// autogenerated by go run gen.go -output fixedhuff.go, DO NOT EDIT")
fmt.Fprintln(&buf)
fmt.Fprintln(&buf, "var fixedHuffmanDecoder = huffmanDecoder{")
fmt.Fprintf(&buf, "\t%d,\n", h.min)
fmt.Fprintln(&buf, "\t[huffmanNumChunks]uint32{")
for i := 0; i < huffmanNumChunks; i++ {
if i&7 == 0 {
fmt.Fprintf(&buf, "\t\t")
} else {
fmt.Fprintf(&buf, " ")
}
fmt.Fprintf(&buf, "0x%04x,", h.chunks[i])
if i&7 == 7 {
fmt.Fprintln(&buf)
}
}
fmt.Fprintln(&buf, "\t},")
fmt.Fprintln(&buf, "\tnil, 0,")
fmt.Fprintln(&buf, "}")
data, err := format.Source(buf.Bytes())
if err != nil {
log.Fatal(err)
}
err = ioutil.WriteFile(*filename, data, 0644)
if err != nil {
log.Fatal(err)
}
}
var reverseByte [256]byte
func initReverseByte() {
for x := 0; x < 256; x++ {
var result byte
for i := uint(0); i < 8; i++ {
result |= byte(((x >> i) & 1) << (7 - i))
}
reverseByte[x] = result
}
}
+701
View File
@@ -0,0 +1,701 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"io"
)
const (
// The largest offset code.
offsetCodeCount = 30
// The special code used to mark the end of a block.
endBlockMarker = 256
// The first length code.
lengthCodesStart = 257
// The number of codegen codes.
codegenCodeCount = 19
badCode = 255
// bufferFlushSize indicates the buffer size
// after which bytes are flushed to the writer.
// Should preferably be a multiple of 6, since
// we accumulate 6 bytes between writes to the buffer.
bufferFlushSize = 240
// bufferSize is the actual output byte buffer size.
// It must have additional headroom for a flush
// which can contain up to 8 bytes.
bufferSize = bufferFlushSize + 8
)
// The number of extra bits needed by length code X - LENGTH_CODES_START.
var lengthExtraBits = []int8{
/* 257 */ 0, 0, 0,
/* 260 */ 0, 0, 0, 0, 0, 1, 1, 1, 1, 2,
/* 270 */ 2, 2, 2, 3, 3, 3, 3, 4, 4, 4,
/* 280 */ 4, 5, 5, 5, 5, 0,
}
// The length indicated by length code X - LENGTH_CODES_START.
var lengthBase = []uint32{
0, 1, 2, 3, 4, 5, 6, 7, 8, 10,
12, 14, 16, 20, 24, 28, 32, 40, 48, 56,
64, 80, 96, 112, 128, 160, 192, 224, 255,
}
// offset code word extra bits.
var offsetExtraBits = []int8{
0, 0, 0, 0, 1, 1, 2, 2, 3, 3,
4, 4, 5, 5, 6, 6, 7, 7, 8, 8,
9, 9, 10, 10, 11, 11, 12, 12, 13, 13,
/* extended window */
14, 14, 15, 15, 16, 16, 17, 17, 18, 18, 19, 19, 20, 20,
}
var offsetBase = []uint32{
/* normal deflate */
0x000000, 0x000001, 0x000002, 0x000003, 0x000004,
0x000006, 0x000008, 0x00000c, 0x000010, 0x000018,
0x000020, 0x000030, 0x000040, 0x000060, 0x000080,
0x0000c0, 0x000100, 0x000180, 0x000200, 0x000300,
0x000400, 0x000600, 0x000800, 0x000c00, 0x001000,
0x001800, 0x002000, 0x003000, 0x004000, 0x006000,
/* extended window */
0x008000, 0x00c000, 0x010000, 0x018000, 0x020000,
0x030000, 0x040000, 0x060000, 0x080000, 0x0c0000,
0x100000, 0x180000, 0x200000, 0x300000,
}
// The odd order in which the codegen code sizes are written.
var codegenOrder = []uint32{16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}
type huffmanBitWriter struct {
// writer is the underlying writer.
// Do not use it directly; use the write method, which ensures
// that Write errors are sticky.
writer io.Writer
// Data waiting to be written is bytes[0:nbytes]
// and then the low nbits of bits.
bits uint64
nbits uint
bytes [bufferSize]byte
codegenFreq [codegenCodeCount]int32
nbytes int
literalFreq []int32
offsetFreq []int32
codegen []uint8
literalEncoding *huffmanEncoder
offsetEncoding *huffmanEncoder
codegenEncoding *huffmanEncoder
err error
}
func newHuffmanBitWriter(w io.Writer) *huffmanBitWriter {
return &huffmanBitWriter{
writer: w,
literalFreq: make([]int32, maxNumLit),
offsetFreq: make([]int32, offsetCodeCount),
codegen: make([]uint8, maxNumLit+offsetCodeCount+1),
literalEncoding: newHuffmanEncoder(maxNumLit),
codegenEncoding: newHuffmanEncoder(codegenCodeCount),
offsetEncoding: newHuffmanEncoder(offsetCodeCount),
}
}
func (w *huffmanBitWriter) reset(writer io.Writer) {
w.writer = writer
w.bits, w.nbits, w.nbytes, w.err = 0, 0, 0, nil
w.bytes = [bufferSize]byte{}
}
func (w *huffmanBitWriter) flush() {
if w.err != nil {
w.nbits = 0
return
}
n := w.nbytes
for w.nbits != 0 {
w.bytes[n] = byte(w.bits)
w.bits >>= 8
if w.nbits > 8 { // Avoid underflow
w.nbits -= 8
} else {
w.nbits = 0
}
n++
}
w.bits = 0
w.write(w.bytes[:n])
w.nbytes = 0
}
func (w *huffmanBitWriter) write(b []byte) {
if w.err != nil {
return
}
_, w.err = w.writer.Write(b)
}
func (w *huffmanBitWriter) writeBits(b int32, nb uint) {
if w.err != nil {
return
}
w.bits |= uint64(b) << w.nbits
w.nbits += nb
if w.nbits >= 48 {
bits := w.bits
w.bits >>= 48
w.nbits -= 48
n := w.nbytes
bytes := w.bytes[n : n+6]
bytes[0] = byte(bits)
bytes[1] = byte(bits >> 8)
bytes[2] = byte(bits >> 16)
bytes[3] = byte(bits >> 24)
bytes[4] = byte(bits >> 32)
bytes[5] = byte(bits >> 40)
n += 6
if n >= bufferFlushSize {
w.write(w.bytes[:n])
n = 0
}
w.nbytes = n
}
}
func (w *huffmanBitWriter) writeBytes(bytes []byte) {
if w.err != nil {
return
}
n := w.nbytes
if w.nbits&7 != 0 {
w.err = InternalError("writeBytes with unfinished bits")
return
}
for w.nbits != 0 {
w.bytes[n] = byte(w.bits)
w.bits >>= 8
w.nbits -= 8
n++
}
if n != 0 {
w.write(w.bytes[:n])
}
w.nbytes = 0
w.write(bytes)
}
// RFC 1951 3.2.7 specifies a special run-length encoding for specifying
// the literal and offset lengths arrays (which are concatenated into a single
// array). This method generates that run-length encoding.
//
// The result is written into the codegen array, and the frequencies
// of each code is written into the codegenFreq array.
// Codes 0-15 are single byte codes. Codes 16-18 are followed by additional
// information. Code badCode is an end marker
//
// numLiterals The number of literals in literalEncoding
// numOffsets The number of offsets in offsetEncoding
// litenc, offenc The literal and offset encoder to use
func (w *huffmanBitWriter) generateCodegen(numLiterals int, numOffsets int, litEnc, offEnc *huffmanEncoder) {
for i := range w.codegenFreq {
w.codegenFreq[i] = 0
}
// Note that we are using codegen both as a temporary variable for holding
// a copy of the frequencies, and as the place where we put the result.
// This is fine because the output is always shorter than the input used
// so far.
codegen := w.codegen // cache
// Copy the concatenated code sizes to codegen. Put a marker at the end.
cgnl := codegen[:numLiterals]
for i := range cgnl {
cgnl[i] = uint8(litEnc.codes[i].len)
}
cgnl = codegen[numLiterals : numLiterals+numOffsets]
for i := range cgnl {
cgnl[i] = uint8(offEnc.codes[i].len)
}
codegen[numLiterals+numOffsets] = badCode
size := codegen[0]
count := 1
outIndex := 0
for inIndex := 1; size != badCode; inIndex++ {
// INVARIANT: We have seen "count" copies of size that have not yet
// had output generated for them.
nextSize := codegen[inIndex]
if nextSize == size {
count++
continue
}
// We need to generate codegen indicating "count" of size.
if size != 0 {
codegen[outIndex] = size
outIndex++
w.codegenFreq[size]++
count--
for count >= 3 {
n := 6
if n > count {
n = count
}
codegen[outIndex] = 16
outIndex++
codegen[outIndex] = uint8(n - 3)
outIndex++
w.codegenFreq[16]++
count -= n
}
} else {
for count >= 11 {
n := 138
if n > count {
n = count
}
codegen[outIndex] = 18
outIndex++
codegen[outIndex] = uint8(n - 11)
outIndex++
w.codegenFreq[18]++
count -= n
}
if count >= 3 {
// count >= 3 && count <= 10
codegen[outIndex] = 17
outIndex++
codegen[outIndex] = uint8(count - 3)
outIndex++
w.codegenFreq[17]++
count = 0
}
}
count--
for ; count >= 0; count-- {
codegen[outIndex] = size
outIndex++
w.codegenFreq[size]++
}
// Set up invariant for next time through the loop.
size = nextSize
count = 1
}
// Marker indicating the end of the codegen.
codegen[outIndex] = badCode
}
// dynamicSize returns the size of dynamically encoded data in bits.
func (w *huffmanBitWriter) dynamicSize(litEnc, offEnc *huffmanEncoder, extraBits int) (size, numCodegens int) {
numCodegens = len(w.codegenFreq)
for numCodegens > 4 && w.codegenFreq[codegenOrder[numCodegens-1]] == 0 {
numCodegens--
}
header := 3 + 5 + 5 + 4 + (3 * numCodegens) +
w.codegenEncoding.bitLength(w.codegenFreq[:]) +
int(w.codegenFreq[16])*2 +
int(w.codegenFreq[17])*3 +
int(w.codegenFreq[18])*7
size = header +
litEnc.bitLength(w.literalFreq) +
offEnc.bitLength(w.offsetFreq) +
extraBits
return size, numCodegens
}
// fixedSize returns the size of dynamically encoded data in bits.
func (w *huffmanBitWriter) fixedSize(extraBits int) int {
return 3 +
fixedLiteralEncoding.bitLength(w.literalFreq) +
fixedOffsetEncoding.bitLength(w.offsetFreq) +
extraBits
}
// storedSize calculates the stored size, including header.
// The function returns the size in bits and whether the block
// fits inside a single block.
func (w *huffmanBitWriter) storedSize(in []byte) (int, bool) {
if in == nil {
return 0, false
}
if len(in) <= maxStoreBlockSize {
return (len(in) + 5) * 8, true
}
return 0, false
}
func (w *huffmanBitWriter) writeCode(c hcode) {
if w.err != nil {
return
}
w.bits |= uint64(c.code) << w.nbits
w.nbits += uint(c.len)
if w.nbits >= 48 {
bits := w.bits
w.bits >>= 48
w.nbits -= 48
n := w.nbytes
bytes := w.bytes[n : n+6]
bytes[0] = byte(bits)
bytes[1] = byte(bits >> 8)
bytes[2] = byte(bits >> 16)
bytes[3] = byte(bits >> 24)
bytes[4] = byte(bits >> 32)
bytes[5] = byte(bits >> 40)
n += 6
if n >= bufferFlushSize {
w.write(w.bytes[:n])
n = 0
}
w.nbytes = n
}
}
// Write the header of a dynamic Huffman block to the output stream.
//
// numLiterals The number of literals specified in codegen
// numOffsets The number of offsets specified in codegen
// numCodegens The number of codegens used in codegen
func (w *huffmanBitWriter) writeDynamicHeader(numLiterals int, numOffsets int, numCodegens int, isEof bool) {
if w.err != nil {
return
}
var firstBits int32 = 4
if isEof {
firstBits = 5
}
w.writeBits(firstBits, 3)
w.writeBits(int32(numLiterals-257), 5)
w.writeBits(int32(numOffsets-1), 5)
w.writeBits(int32(numCodegens-4), 4)
for i := 0; i < numCodegens; i++ {
value := uint(w.codegenEncoding.codes[codegenOrder[i]].len)
w.writeBits(int32(value), 3)
}
i := 0
for {
var codeWord int = int(w.codegen[i])
i++
if codeWord == badCode {
break
}
w.writeCode(w.codegenEncoding.codes[uint32(codeWord)])
switch codeWord {
case 16:
w.writeBits(int32(w.codegen[i]), 2)
i++
break
case 17:
w.writeBits(int32(w.codegen[i]), 3)
i++
break
case 18:
w.writeBits(int32(w.codegen[i]), 7)
i++
break
}
}
}
func (w *huffmanBitWriter) writeStoredHeader(length int, isEof bool) {
if w.err != nil {
return
}
var flag int32
if isEof {
flag = 1
}
w.writeBits(flag, 3)
w.flush()
w.writeBits(int32(length), 16)
w.writeBits(int32(^uint16(length)), 16)
}
func (w *huffmanBitWriter) writeFixedHeader(isEof bool) {
if w.err != nil {
return
}
// Indicate that we are a fixed Huffman block
var value int32 = 2
if isEof {
value = 3
}
w.writeBits(value, 3)
}
// writeBlock will write a block of tokens with the smallest encoding.
// The original input can be supplied, and if the huffman encoded data
// is larger than the original bytes, the data will be written as a
// stored block.
// If the input is nil, the tokens will always be Huffman encoded.
func (w *huffmanBitWriter) writeBlock(tokens []token, eof bool, input []byte) {
if w.err != nil {
return
}
tokens = append(tokens, endBlockMarker)
numLiterals, numOffsets := w.indexTokens(tokens)
var extraBits int
storedSize, storable := w.storedSize(input)
if storable {
// We only bother calculating the costs of the extra bits required by
// the length of offset fields (which will be the same for both fixed
// and dynamic encoding), if we need to compare those two encodings
// against stored encoding.
for lengthCode := lengthCodesStart + 8; lengthCode < numLiterals; lengthCode++ {
// First eight length codes have extra size = 0.
extraBits += int(w.literalFreq[lengthCode]) * int(lengthExtraBits[lengthCode-lengthCodesStart])
}
for offsetCode := 4; offsetCode < numOffsets; offsetCode++ {
// First four offset codes have extra size = 0.
extraBits += int(w.offsetFreq[offsetCode]) * int(offsetExtraBits[offsetCode])
}
}
// Figure out smallest code.
// Fixed Huffman baseline.
var literalEncoding = fixedLiteralEncoding
var offsetEncoding = fixedOffsetEncoding
var size = w.fixedSize(extraBits)
// Dynamic Huffman?
var numCodegens int
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, w.literalEncoding, w.offsetEncoding)
w.codegenEncoding.generate(w.codegenFreq[:], 7)
dynamicSize, numCodegens := w.dynamicSize(w.literalEncoding, w.offsetEncoding, extraBits)
if dynamicSize < size {
size = dynamicSize
literalEncoding = w.literalEncoding
offsetEncoding = w.offsetEncoding
}
// Stored bytes?
if storable && storedSize < size {
w.writeStoredHeader(len(input), eof)
w.writeBytes(input)
return
}
// Huffman.
if literalEncoding == fixedLiteralEncoding {
w.writeFixedHeader(eof)
} else {
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
}
// Write the tokens.
w.writeTokens(tokens, literalEncoding.codes, offsetEncoding.codes)
}
// writeBlockDynamic encodes a block using a dynamic Huffman table.
// This should be used if the symbols used have a disproportionate
// histogram distribution.
// If input is supplied and the compression savings are below 1/16th of the
// input size the block is stored.
func (w *huffmanBitWriter) writeBlockDynamic(tokens []token, eof bool, input []byte) {
if w.err != nil {
return
}
tokens = append(tokens, endBlockMarker)
numLiterals, numOffsets := w.indexTokens(tokens)
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, w.literalEncoding, w.offsetEncoding)
w.codegenEncoding.generate(w.codegenFreq[:], 7)
size, numCodegens := w.dynamicSize(w.literalEncoding, w.offsetEncoding, 0)
// Store bytes, if we don't get a reasonable improvement.
if ssize, storable := w.storedSize(input); storable && ssize < (size+size>>4) {
w.writeStoredHeader(len(input), eof)
w.writeBytes(input)
return
}
// Write Huffman table.
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
// Write the tokens.
w.writeTokens(tokens, w.literalEncoding.codes, w.offsetEncoding.codes)
}
// indexTokens indexes a slice of tokens, and updates
// literalFreq and offsetFreq, and generates literalEncoding
// and offsetEncoding.
// The number of literal and offset tokens is returned.
func (w *huffmanBitWriter) indexTokens(tokens []token) (numLiterals, numOffsets int) {
for i := range w.literalFreq {
w.literalFreq[i] = 0
}
for i := range w.offsetFreq {
w.offsetFreq[i] = 0
}
for _, t := range tokens {
if t < matchType {
w.literalFreq[t.literal()]++
continue
}
length := t.length()
offset := t.offset()
w.literalFreq[lengthCodesStart+lengthCode(length)]++
w.offsetFreq[offsetCode(offset)]++
}
// get the number of literals
numLiterals = len(w.literalFreq)
for w.literalFreq[numLiterals-1] == 0 {
numLiterals--
}
// get the number of offsets
numOffsets = len(w.offsetFreq)
for numOffsets > 0 && w.offsetFreq[numOffsets-1] == 0 {
numOffsets--
}
if numOffsets == 0 {
// We haven't found a single match. If we want to go with the dynamic encoding,
// we should count at least one offset to be sure that the offset huffman tree could be encoded.
w.offsetFreq[0] = 1
numOffsets = 1
}
w.literalEncoding.generate(w.literalFreq, 15)
w.offsetEncoding.generate(w.offsetFreq, 15)
return
}
// writeTokens writes a slice of tokens to the output.
// codes for literal and offset encoding must be supplied.
func (w *huffmanBitWriter) writeTokens(tokens []token, leCodes, oeCodes []hcode) {
if w.err != nil {
return
}
for _, t := range tokens {
if t < matchType {
w.writeCode(leCodes[t.literal()])
continue
}
// Write the length
length := t.length()
lengthCode := lengthCode(length)
w.writeCode(leCodes[lengthCode+lengthCodesStart])
extraLengthBits := uint(lengthExtraBits[lengthCode])
if extraLengthBits > 0 {
extraLength := int32(length - lengthBase[lengthCode])
w.writeBits(extraLength, extraLengthBits)
}
// Write the offset
offset := t.offset()
offsetCode := offsetCode(offset)
w.writeCode(oeCodes[offsetCode])
extraOffsetBits := uint(offsetExtraBits[offsetCode])
if extraOffsetBits > 0 {
extraOffset := int32(offset - offsetBase[offsetCode])
w.writeBits(extraOffset, extraOffsetBits)
}
}
}
// huffOffset is a static offset encoder used for huffman only encoding.
// It can be reused since we will not be encoding offset values.
var huffOffset *huffmanEncoder
func init() {
w := newHuffmanBitWriter(nil)
w.offsetFreq[0] = 1
huffOffset = newHuffmanEncoder(offsetCodeCount)
huffOffset.generate(w.offsetFreq, 15)
}
// writeBlockHuff encodes a block of bytes as either
// Huffman encoded literals or uncompressed bytes if the
// results only gains very little from compression.
func (w *huffmanBitWriter) writeBlockHuff(eof bool, input []byte) {
if w.err != nil {
return
}
// Clear histogram
for i := range w.literalFreq {
w.literalFreq[i] = 0
}
// Add everything as literals
histogram(input, w.literalFreq)
w.literalFreq[endBlockMarker] = 1
const numLiterals = endBlockMarker + 1
const numOffsets = 1
w.literalEncoding.generate(w.literalFreq, 15)
// Figure out smallest code.
// Always use dynamic Huffman or Store
var numCodegens int
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, w.literalEncoding, huffOffset)
w.codegenEncoding.generate(w.codegenFreq[:], 7)
size, numCodegens := w.dynamicSize(w.literalEncoding, huffOffset, 0)
// Store bytes, if we don't get a reasonable improvement.
if ssize, storable := w.storedSize(input); storable && ssize < (size+size>>4) {
w.writeStoredHeader(len(input), eof)
w.writeBytes(input)
return
}
// Huffman.
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
encoding := w.literalEncoding.codes[:257]
n := w.nbytes
for _, t := range input {
// Bitwriting inlined, ~30% speedup
c := encoding[t]
w.bits |= uint64(c.code) << w.nbits
w.nbits += uint(c.len)
if w.nbits < 48 {
continue
}
// Store 6 bytes
bits := w.bits
w.bits >>= 48
w.nbits -= 48
bytes := w.bytes[n : n+6]
bytes[0] = byte(bits)
bytes[1] = byte(bits >> 8)
bytes[2] = byte(bits >> 16)
bytes[3] = byte(bits >> 24)
bytes[4] = byte(bits >> 32)
bytes[5] = byte(bits >> 40)
n += 6
if n < bufferFlushSize {
continue
}
w.write(w.bytes[:n])
if w.err != nil {
return // Return early in the event of write failures
}
n = 0
}
w.nbytes = n
w.writeCode(encoding[endBlockMarker])
}
+344
View File
@@ -0,0 +1,344 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"math"
"sort"
)
// hcode is a huffman code with a bit code and bit length.
type hcode struct {
code, len uint16
}
type huffmanEncoder struct {
codes []hcode
freqcache []literalNode
bitCount [17]int32
lns byLiteral // stored to avoid repeated allocation in generate
lfs byFreq // stored to avoid repeated allocation in generate
}
type literalNode struct {
literal uint16
freq int32
}
// A levelInfo describes the state of the constructed tree for a given depth.
type levelInfo struct {
// Our level. for better printing
level int32
// The frequency of the last node at this level
lastFreq int32
// The frequency of the next character to add to this level
nextCharFreq int32
// The frequency of the next pair (from level below) to add to this level.
// Only valid if the "needed" value of the next lower level is 0.
nextPairFreq int32
// The number of chains remaining to generate for this level before moving
// up to the next level
needed int32
}
// set sets the code and length of an hcode.
func (h *hcode) set(code uint16, length uint16) {
h.len = length
h.code = code
}
func maxNode() literalNode { return literalNode{math.MaxUint16, math.MaxInt32} }
func newHuffmanEncoder(size int) *huffmanEncoder {
return &huffmanEncoder{codes: make([]hcode, size)}
}
// Generates a HuffmanCode corresponding to the fixed literal table
func generateFixedLiteralEncoding() *huffmanEncoder {
h := newHuffmanEncoder(maxNumLit)
codes := h.codes
var ch uint16
for ch = 0; ch < maxNumLit; ch++ {
var bits uint16
var size uint16
switch {
case ch < 144:
// size 8, 000110000 .. 10111111
bits = ch + 48
size = 8
break
case ch < 256:
// size 9, 110010000 .. 111111111
bits = ch + 400 - 144
size = 9
break
case ch < 280:
// size 7, 0000000 .. 0010111
bits = ch - 256
size = 7
break
default:
// size 8, 11000000 .. 11000111
bits = ch + 192 - 280
size = 8
}
codes[ch] = hcode{code: reverseBits(bits, byte(size)), len: size}
}
return h
}
func generateFixedOffsetEncoding() *huffmanEncoder {
h := newHuffmanEncoder(30)
codes := h.codes
for ch := range codes {
codes[ch] = hcode{code: reverseBits(uint16(ch), 5), len: 5}
}
return h
}
var fixedLiteralEncoding *huffmanEncoder = generateFixedLiteralEncoding()
var fixedOffsetEncoding *huffmanEncoder = generateFixedOffsetEncoding()
func (h *huffmanEncoder) bitLength(freq []int32) int {
var total int
for i, f := range freq {
if f != 0 {
total += int(f) * int(h.codes[i].len)
}
}
return total
}
const maxBitsLimit = 16
// Return the number of literals assigned to each bit size in the Huffman encoding
//
// This method is only called when list.length >= 3
// The cases of 0, 1, and 2 literals are handled by special case code.
//
// list An array of the literals with non-zero frequencies
// and their associated frequencies. The array is in order of increasing
// frequency, and has as its last element a special element with frequency
// MaxInt32
// maxBits The maximum number of bits that should be used to encode any literal.
// Must be less than 16.
// return An integer array in which array[i] indicates the number of literals
// that should be encoded in i bits.
func (h *huffmanEncoder) bitCounts(list []literalNode, maxBits int32) []int32 {
if maxBits >= maxBitsLimit {
panic("flate: maxBits too large")
}
n := int32(len(list))
list = list[0 : n+1]
list[n] = maxNode()
// The tree can't have greater depth than n - 1, no matter what. This
// saves a little bit of work in some small cases
if maxBits > n-1 {
maxBits = n - 1
}
// Create information about each of the levels.
// A bogus "Level 0" whose sole purpose is so that
// level1.prev.needed==0. This makes level1.nextPairFreq
// be a legitimate value that never gets chosen.
var levels [maxBitsLimit]levelInfo
// leafCounts[i] counts the number of literals at the left
// of ancestors of the rightmost node at level i.
// leafCounts[i][j] is the number of literals at the left
// of the level j ancestor.
var leafCounts [maxBitsLimit][maxBitsLimit]int32
for level := int32(1); level <= maxBits; level++ {
// For every level, the first two items are the first two characters.
// We initialize the levels as if we had already figured this out.
levels[level] = levelInfo{
level: level,
lastFreq: list[1].freq,
nextCharFreq: list[2].freq,
nextPairFreq: list[0].freq + list[1].freq,
}
leafCounts[level][level] = 2
if level == 1 {
levels[level].nextPairFreq = math.MaxInt32
}
}
// We need a total of 2*n - 2 items at top level and have already generated 2.
levels[maxBits].needed = 2*n - 4
level := maxBits
for {
l := &levels[level]
if l.nextPairFreq == math.MaxInt32 && l.nextCharFreq == math.MaxInt32 {
// We've run out of both leafs and pairs.
// End all calculations for this level.
// To make sure we never come back to this level or any lower level,
// set nextPairFreq impossibly large.
l.needed = 0
levels[level+1].nextPairFreq = math.MaxInt32
level++
continue
}
prevFreq := l.lastFreq
if l.nextCharFreq < l.nextPairFreq {
// The next item on this row is a leaf node.
n := leafCounts[level][level] + 1
l.lastFreq = l.nextCharFreq
// Lower leafCounts are the same of the previous node.
leafCounts[level][level] = n
l.nextCharFreq = list[n].freq
} else {
// The next item on this row is a pair from the previous row.
// nextPairFreq isn't valid until we generate two
// more values in the level below
l.lastFreq = l.nextPairFreq
// Take leaf counts from the lower level, except counts[level] remains the same.
copy(leafCounts[level][:level], leafCounts[level-1][:level])
levels[l.level-1].needed = 2
}
if l.needed--; l.needed == 0 {
// We've done everything we need to do for this level.
// Continue calculating one level up. Fill in nextPairFreq
// of that level with the sum of the two nodes we've just calculated on
// this level.
if l.level == maxBits {
// All done!
break
}
levels[l.level+1].nextPairFreq = prevFreq + l.lastFreq
level++
} else {
// If we stole from below, move down temporarily to replenish it.
for levels[level-1].needed > 0 {
level--
}
}
}
// Somethings is wrong if at the end, the top level is null or hasn't used
// all of the leaves.
if leafCounts[maxBits][maxBits] != n {
panic("leafCounts[maxBits][maxBits] != n")
}
bitCount := h.bitCount[:maxBits+1]
bits := 1
counts := &leafCounts[maxBits]
for level := maxBits; level > 0; level-- {
// chain.leafCount gives the number of literals requiring at least "bits"
// bits to encode.
bitCount[bits] = counts[level] - counts[level-1]
bits++
}
return bitCount
}
// Look at the leaves and assign them a bit count and an encoding as specified
// in RFC 1951 3.2.2
func (h *huffmanEncoder) assignEncodingAndSize(bitCount []int32, list []literalNode) {
code := uint16(0)
for n, bits := range bitCount {
code <<= 1
if n == 0 || bits == 0 {
continue
}
// The literals list[len(list)-bits] .. list[len(list)-bits]
// are encoded using "bits" bits, and get the values
// code, code + 1, .... The code values are
// assigned in literal order (not frequency order).
chunk := list[len(list)-int(bits):]
h.lns.sort(chunk)
for _, node := range chunk {
h.codes[node.literal] = hcode{code: reverseBits(code, uint8(n)), len: uint16(n)}
code++
}
list = list[0 : len(list)-int(bits)]
}
}
// Update this Huffman Code object to be the minimum code for the specified frequency count.
//
// freq An array of frequencies, in which frequency[i] gives the frequency of literal i.
// maxBits The maximum number of bits to use for any literal.
func (h *huffmanEncoder) generate(freq []int32, maxBits int32) {
if h.freqcache == nil {
// Allocate a reusable buffer with the longest possible frequency table.
// Possible lengths are codegenCodeCount, offsetCodeCount and maxNumLit.
// The largest of these is maxNumLit, so we allocate for that case.
h.freqcache = make([]literalNode, maxNumLit+1)
}
list := h.freqcache[:len(freq)+1]
// Number of non-zero literals
count := 0
// Set list to be the set of all non-zero literals and their frequencies
for i, f := range freq {
if f != 0 {
list[count] = literalNode{uint16(i), f}
count++
} else {
list[count] = literalNode{}
h.codes[i].len = 0
}
}
list[len(freq)] = literalNode{}
list = list[:count]
if count <= 2 {
// Handle the small cases here, because they are awkward for the general case code. With
// two or fewer literals, everything has bit length 1.
for i, node := range list {
// "list" is in order of increasing literal value.
h.codes[node.literal].set(uint16(i), 1)
}
return
}
h.lfs.sort(list)
// Get the number of literals for each bit count
bitCount := h.bitCounts(list, maxBits)
// And do the assignment
h.assignEncodingAndSize(bitCount, list)
}
type byLiteral []literalNode
func (s *byLiteral) sort(a []literalNode) {
*s = byLiteral(a)
sort.Sort(s)
}
func (s byLiteral) Len() int { return len(s) }
func (s byLiteral) Less(i, j int) bool {
return s[i].literal < s[j].literal
}
func (s byLiteral) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
type byFreq []literalNode
func (s *byFreq) sort(a []literalNode) {
*s = byFreq(a)
sort.Sort(s)
}
func (s byFreq) Len() int { return len(s) }
func (s byFreq) Less(i, j int) bool {
if s[i].freq == s[j].freq {
return s[i].literal < s[j].literal
}
return s[i].freq < s[j].freq
}
func (s byFreq) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
+868
View File
@@ -0,0 +1,868 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package flate implements the DEFLATE compressed data format, described in
// RFC 1951. The gzip and zlib packages implement access to DEFLATE-based file
// formats.
package flate
import (
"bufio"
"io"
"strconv"
"sync"
)
const (
maxCodeLen = 16 // max length of Huffman code
maxCodeLenMask = 15 // mask for max length of Huffman code
// The next three numbers come from the RFC section 3.2.7, with the
// additional proviso in section 3.2.5 which implies that distance codes
// 30 and 31 should never occur in compressed data.
maxNumLit = 286
maxNumDist = 30
numCodes = 19 // number of codes in Huffman meta-code
)
// Initialize the fixedHuffmanDecoder only once upon first use.
var fixedOnce sync.Once
var fixedHuffmanDecoder huffmanDecoder
// A CorruptInputError reports the presence of corrupt input at a given offset.
type CorruptInputError int64
func (e CorruptInputError) Error() string {
return "flate: corrupt input before offset " + strconv.FormatInt(int64(e), 10)
}
// An InternalError reports an error in the flate code itself.
type InternalError string
func (e InternalError) Error() string { return "flate: internal error: " + string(e) }
// A ReadError reports an error encountered while reading input.
//
// Deprecated: No longer returned.
type ReadError struct {
Offset int64 // byte offset where error occurred
Err error // error returned by underlying Read
}
func (e *ReadError) Error() string {
return "flate: read error at offset " + strconv.FormatInt(e.Offset, 10) + ": " + e.Err.Error()
}
// A WriteError reports an error encountered while writing output.
//
// Deprecated: No longer returned.
type WriteError struct {
Offset int64 // byte offset where error occurred
Err error // error returned by underlying Write
}
func (e *WriteError) Error() string {
return "flate: write error at offset " + strconv.FormatInt(e.Offset, 10) + ": " + e.Err.Error()
}
// Resetter resets a ReadCloser returned by NewReader or NewReaderDict to
// to switch to a new underlying Reader. This permits reusing a ReadCloser
// instead of allocating a new one.
type Resetter interface {
// Reset discards any buffered data and resets the Resetter as if it was
// newly initialized with the given reader.
Reset(r io.Reader, dict []byte) error
}
// The data structure for decoding Huffman tables is based on that of
// zlib. There is a lookup table of a fixed bit width (huffmanChunkBits),
// For codes smaller than the table width, there are multiple entries
// (each combination of trailing bits has the same value). For codes
// larger than the table width, the table contains a link to an overflow
// table. The width of each entry in the link table is the maximum code
// size minus the chunk width.
//
// Note that you can do a lookup in the table even without all bits
// filled. Since the extra bits are zero, and the DEFLATE Huffman codes
// have the property that shorter codes come before longer ones, the
// bit length estimate in the result is a lower bound on the actual
// number of bits.
//
// See the following:
// http://www.gzip.org/algorithm.txt
// chunk & 15 is number of bits
// chunk >> 4 is value, including table link
const (
huffmanChunkBits = 9
huffmanNumChunks = 1 << huffmanChunkBits
huffmanCountMask = 15
huffmanValueShift = 4
)
type huffmanDecoder struct {
min int // the minimum code length
chunks *[huffmanNumChunks]uint32 // chunks as described above
links [][]uint32 // overflow links
linkMask uint32 // mask the width of the link table
}
// Initialize Huffman decoding tables from array of code lengths.
// Following this function, h is guaranteed to be initialized into a complete
// tree (i.e., neither over-subscribed nor under-subscribed). The exception is a
// degenerate case where the tree has only a single symbol with length 1. Empty
// trees are permitted.
func (h *huffmanDecoder) init(bits []int) bool {
// Sanity enables additional runtime tests during Huffman
// table construction. It's intended to be used during
// development to supplement the currently ad-hoc unit tests.
const sanity = false
if h.chunks == nil {
h.chunks = &[huffmanNumChunks]uint32{}
}
if h.min != 0 {
*h = huffmanDecoder{chunks: h.chunks, links: h.links}
}
// Count number of codes of each length,
// compute min and max length.
var count [maxCodeLen]int
var min, max int
for _, n := range bits {
if n == 0 {
continue
}
if min == 0 || n < min {
min = n
}
if n > max {
max = n
}
count[n&maxCodeLenMask]++
}
// Empty tree. The decompressor.huffSym function will fail later if the tree
// is used. Technically, an empty tree is only valid for the HDIST tree and
// not the HCLEN and HLIT tree. However, a stream with an empty HCLEN tree
// is guaranteed to fail since it will attempt to use the tree to decode the
// codes for the HLIT and HDIST trees. Similarly, an empty HLIT tree is
// guaranteed to fail later since the compressed data section must be
// composed of at least one symbol (the end-of-block marker).
if max == 0 {
return true
}
code := 0
var nextcode [maxCodeLen]int
for i := min; i <= max; i++ {
code <<= 1
nextcode[i&maxCodeLenMask] = code
code += count[i&maxCodeLenMask]
}
// Check that the coding is complete (i.e., that we've
// assigned all 2-to-the-max possible bit sequences).
// Exception: To be compatible with zlib, we also need to
// accept degenerate single-code codings. See also
// TestDegenerateHuffmanCoding.
if code != 1<<uint(max) && !(code == 1 && max == 1) {
return false
}
h.min = min
chunks := h.chunks[:]
for i := range chunks {
chunks[i] = 0
}
if max > huffmanChunkBits {
numLinks := 1 << (uint(max) - huffmanChunkBits)
h.linkMask = uint32(numLinks - 1)
// create link tables
link := nextcode[huffmanChunkBits+1] >> 1
if cap(h.links) < huffmanNumChunks-link {
h.links = make([][]uint32, huffmanNumChunks-link)
} else {
h.links = h.links[:huffmanNumChunks-link]
}
for j := uint(link); j < huffmanNumChunks; j++ {
reverse := int(reverseByte[j>>8]) | int(reverseByte[j&0xff])<<8
reverse >>= uint(16 - huffmanChunkBits)
off := j - uint(link)
if sanity && h.chunks[reverse] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[reverse] = uint32(off<<huffmanValueShift | (huffmanChunkBits + 1))
if cap(h.links[off]) < numLinks {
h.links[off] = make([]uint32, numLinks)
} else {
links := h.links[off][:0]
h.links[off] = links[:numLinks]
}
}
} else {
h.links = h.links[:0]
}
for i, n := range bits {
if n == 0 {
continue
}
code := nextcode[n]
nextcode[n]++
chunk := uint32(i<<huffmanValueShift | n)
reverse := int(reverseByte[code>>8]) | int(reverseByte[code&0xff])<<8
reverse >>= uint(16 - n)
if n <= huffmanChunkBits {
for off := reverse; off < len(h.chunks); off += 1 << uint(n) {
// We should never need to overwrite
// an existing chunk. Also, 0 is
// never a valid chunk, because the
// lower 4 "count" bits should be
// between 1 and 15.
if sanity && h.chunks[off] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[off] = chunk
}
} else {
j := reverse & (huffmanNumChunks - 1)
if sanity && h.chunks[j]&huffmanCountMask != huffmanChunkBits+1 {
// Longer codes should have been
// associated with a link table above.
panic("impossible: not an indirect chunk")
}
value := h.chunks[j] >> huffmanValueShift
linktab := h.links[value]
reverse >>= huffmanChunkBits
for off := reverse; off < len(linktab); off += 1 << uint(n-huffmanChunkBits) {
if sanity && linktab[off] != 0 {
panic("impossible: overwriting existing chunk")
}
linktab[off] = chunk
}
}
}
if sanity {
// Above we've sanity checked that we never overwrote
// an existing entry. Here we additionally check that
// we filled the tables completely.
for i, chunk := range h.chunks {
if chunk == 0 {
// As an exception, in the degenerate
// single-code case, we allow odd
// chunks to be missing.
if code == 1 && i%2 == 1 {
continue
}
panic("impossible: missing chunk")
}
}
for _, linktab := range h.links {
for _, chunk := range linktab {
if chunk == 0 {
panic("impossible: missing chunk")
}
}
}
}
return true
}
// The actual read interface needed by NewReader.
// If the passed in io.Reader does not also have ReadByte,
// the NewReader will introduce its own buffering.
type Reader interface {
io.Reader
io.ByteReader
}
// Decompress state.
type decompressor struct {
// Input source.
r Reader
roffset int64
// Input bits, in top of b.
b uint32
nb uint
// Huffman decoders for literal/length, distance.
h1, h2 huffmanDecoder
// Length arrays used to define Huffman codes.
bits *[maxNumLit + maxNumDist]int
codebits *[numCodes]int
// Output history, buffer.
dict dictDecoder
// Temporary buffer (avoids repeated allocation).
buf [4]byte
// Next step in the decompression,
// and decompression state.
step func(*decompressor)
stepState int
final bool
err error
toRead []byte
hl, hd *huffmanDecoder
copyLen int
copyDist int
}
func (f *decompressor) nextBlock() {
for f.nb < 1+2 {
if f.err = f.moreBits(); f.err != nil {
return
}
}
f.final = f.b&1 == 1
f.b >>= 1
typ := f.b & 3
f.b >>= 2
f.nb -= 1 + 2
switch typ {
case 0:
f.dataBlock()
case 1:
// compressed, fixed Huffman tables
f.hl = &fixedHuffmanDecoder
f.hd = nil
f.huffmanBlock()
case 2:
// compressed, dynamic Huffman tables
if f.err = f.readHuffman(); f.err != nil {
break
}
f.hl = &f.h1
f.hd = &f.h2
f.huffmanBlock()
default:
// 3 is reserved.
f.err = CorruptInputError(f.roffset)
}
}
func (f *decompressor) Read(b []byte) (int, error) {
for {
if len(f.toRead) > 0 {
n := copy(b, f.toRead)
f.toRead = f.toRead[n:]
if len(f.toRead) == 0 {
return n, f.err
}
return n, nil
}
if f.err != nil {
return 0, f.err
}
f.step(f)
if f.err != nil && len(f.toRead) == 0 {
f.toRead = f.dict.readFlush() // Flush what's left in case of error
}
}
}
// Support the io.WriteTo interface for io.Copy and friends.
func (f *decompressor) WriteTo(w io.Writer) (int64, error) {
total := int64(0)
flushed := false
for {
if len(f.toRead) > 0 {
n, err := w.Write(f.toRead)
total += int64(n)
if err != nil {
f.err = err
return total, err
}
if n != len(f.toRead) {
return total, io.ErrShortWrite
}
f.toRead = f.toRead[:0]
}
if f.err != nil && flushed {
if f.err == io.EOF {
return total, nil
}
return total, f.err
}
if f.err == nil {
f.step(f)
}
if len(f.toRead) == 0 && f.err != nil && !flushed {
f.toRead = f.dict.readFlush() // Flush what's left in case of error
flushed = true
}
}
}
func (f *decompressor) Close() error {
if f.err == io.EOF {
return nil
}
return f.err
}
// RFC 1951 section 3.2.7.
// Compression with dynamic Huffman codes
var codeOrder = [...]int{16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}
func (f *decompressor) readHuffman() error {
// HLIT[5], HDIST[5], HCLEN[4].
for f.nb < 5+5+4 {
if err := f.moreBits(); err != nil {
return err
}
}
nlit := int(f.b&0x1F) + 257
if nlit > maxNumLit {
return CorruptInputError(f.roffset)
}
f.b >>= 5
ndist := int(f.b&0x1F) + 1
if ndist > maxNumDist {
return CorruptInputError(f.roffset)
}
f.b >>= 5
nclen := int(f.b&0xF) + 4
// numCodes is 19, so nclen is always valid.
f.b >>= 4
f.nb -= 5 + 5 + 4
// (HCLEN+4)*3 bits: code lengths in the magic codeOrder order.
for i := 0; i < nclen; i++ {
for f.nb < 3 {
if err := f.moreBits(); err != nil {
return err
}
}
f.codebits[codeOrder[i]] = int(f.b & 0x7)
f.b >>= 3
f.nb -= 3
}
for i := nclen; i < len(codeOrder); i++ {
f.codebits[codeOrder[i]] = 0
}
if !f.h1.init(f.codebits[0:]) {
return CorruptInputError(f.roffset)
}
// HLIT + 257 code lengths, HDIST + 1 code lengths,
// using the code length Huffman code.
for i, n := 0, nlit+ndist; i < n; {
x, err := f.huffSym(&f.h1)
if err != nil {
return err
}
if x < 16 {
// Actual length.
f.bits[i] = x
i++
continue
}
// Repeat previous length or zero.
var rep int
var nb uint
var b int
switch x {
default:
return InternalError("unexpected length code")
case 16:
rep = 3
nb = 2
if i == 0 {
return CorruptInputError(f.roffset)
}
b = f.bits[i-1]
case 17:
rep = 3
nb = 3
b = 0
case 18:
rep = 11
nb = 7
b = 0
}
for f.nb < nb {
if err := f.moreBits(); err != nil {
return err
}
}
rep += int(f.b & uint32(1<<nb-1))
f.b >>= nb
f.nb -= nb
if i+rep > n {
return CorruptInputError(f.roffset)
}
for j := 0; j < rep; j++ {
f.bits[i] = b
i++
}
}
if !f.h1.init(f.bits[0:nlit]) || !f.h2.init(f.bits[nlit:nlit+ndist]) {
return CorruptInputError(f.roffset)
}
// As an optimization, we can initialize the min bits to read at a time
// for the HLIT tree to the length of the EOB marker since we know that
// every block must terminate with one. This preserves the property that
// we never read any extra bytes after the end of the DEFLATE stream.
if f.h1.min < f.bits[endBlockMarker] {
f.h1.min = f.bits[endBlockMarker]
}
return nil
}
// Decode a single Huffman block from f.
// hl and hd are the Huffman states for the lit/length values
// and the distance values, respectively. If hd == nil, using the
// fixed distance encoding associated with fixed Huffman blocks.
func (f *decompressor) huffmanBlock() {
const (
stateInit = iota // Zero value must be stateInit
stateDict
)
switch f.stepState {
case stateInit:
goto readLiteral
case stateDict:
goto copyHistory
}
readLiteral:
// Read literal and/or (length, distance) according to RFC section 3.2.3.
{
v, err := f.huffSym(f.hl)
if err != nil {
f.err = err
return
}
var n uint // number of bits extra
var length int
switch {
case v < 256:
f.dict.writeByte(byte(v))
if f.dict.availWrite() == 0 {
f.toRead = f.dict.readFlush()
f.step = (*decompressor).huffmanBlock
f.stepState = stateInit
return
}
goto readLiteral
case v == 256:
f.finishBlock()
return
// otherwise, reference to older data
case v < 265:
length = v - (257 - 3)
n = 0
case v < 269:
length = v*2 - (265*2 - 11)
n = 1
case v < 273:
length = v*4 - (269*4 - 19)
n = 2
case v < 277:
length = v*8 - (273*8 - 35)
n = 3
case v < 281:
length = v*16 - (277*16 - 67)
n = 4
case v < 285:
length = v*32 - (281*32 - 131)
n = 5
case v < maxNumLit:
length = 258
n = 0
default:
f.err = CorruptInputError(f.roffset)
return
}
if n > 0 {
for f.nb < n {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
length += int(f.b & uint32(1<<n-1))
f.b >>= n
f.nb -= n
}
var dist int
if f.hd == nil {
for f.nb < 5 {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
dist = int(reverseByte[(f.b&0x1F)<<3])
f.b >>= 5
f.nb -= 5
} else {
if dist, err = f.huffSym(f.hd); err != nil {
f.err = err
return
}
}
switch {
case dist < 4:
dist++
case dist < maxNumDist:
nb := uint(dist-2) >> 1
// have 1 bit in bottom of dist, need nb more.
extra := (dist & 1) << nb
for f.nb < nb {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
extra |= int(f.b & uint32(1<<nb-1))
f.b >>= nb
f.nb -= nb
dist = 1<<(nb+1) + 1 + extra
default:
f.err = CorruptInputError(f.roffset)
return
}
// No check on length; encoding can be prescient.
if dist > f.dict.histSize() {
f.err = CorruptInputError(f.roffset)
return
}
f.copyLen, f.copyDist = length, dist
goto copyHistory
}
copyHistory:
// Perform a backwards copy according to RFC section 3.2.3.
{
cnt := f.dict.tryWriteCopy(f.copyDist, f.copyLen)
if cnt == 0 {
cnt = f.dict.writeCopy(f.copyDist, f.copyLen)
}
f.copyLen -= cnt
if f.dict.availWrite() == 0 || f.copyLen > 0 {
f.toRead = f.dict.readFlush()
f.step = (*decompressor).huffmanBlock // We need to continue this work
f.stepState = stateDict
return
}
goto readLiteral
}
}
// Copy a single uncompressed data block from input to output.
func (f *decompressor) dataBlock() {
// Uncompressed.
// Discard current half-byte.
f.nb = 0
f.b = 0
// Length then ones-complement of length.
nr, err := io.ReadFull(f.r, f.buf[0:4])
f.roffset += int64(nr)
if err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
f.err = err
return
}
n := int(f.buf[0]) | int(f.buf[1])<<8
nn := int(f.buf[2]) | int(f.buf[3])<<8
if uint16(nn) != uint16(^n) {
f.err = CorruptInputError(f.roffset)
return
}
if n == 0 {
f.toRead = f.dict.readFlush()
f.finishBlock()
return
}
f.copyLen = n
f.copyData()
}
// copyData copies f.copyLen bytes from the underlying reader into f.hist.
// It pauses for reads when f.hist is full.
func (f *decompressor) copyData() {
buf := f.dict.writeSlice()
if len(buf) > f.copyLen {
buf = buf[:f.copyLen]
}
cnt, err := io.ReadFull(f.r, buf)
f.roffset += int64(cnt)
f.copyLen -= cnt
f.dict.writeMark(cnt)
if err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
f.err = err
return
}
if f.dict.availWrite() == 0 || f.copyLen > 0 {
f.toRead = f.dict.readFlush()
f.step = (*decompressor).copyData
return
}
f.finishBlock()
}
func (f *decompressor) finishBlock() {
if f.final {
if f.dict.availRead() > 0 {
f.toRead = f.dict.readFlush()
}
f.err = io.EOF
}
f.step = (*decompressor).nextBlock
}
func (f *decompressor) moreBits() error {
c, err := f.r.ReadByte()
if err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
return err
}
f.roffset++
f.b |= uint32(c) << f.nb
f.nb += 8
return nil
}
// Read the next Huffman-encoded symbol from f according to h.
func (f *decompressor) huffSym(h *huffmanDecoder) (int, error) {
// Since a huffmanDecoder can be empty or be composed of a degenerate tree
// with single element, huffSym must error on these two edge cases. In both
// cases, the chunks slice will be 0 for the invalid sequence, leading it
// satisfy the n == 0 check below.
n := uint(h.min)
for {
for f.nb < n {
if err := f.moreBits(); err != nil {
return 0, err
}
}
chunk := h.chunks[f.b&(huffmanNumChunks-1)]
n = uint(chunk & huffmanCountMask)
if n > huffmanChunkBits {
chunk = h.links[chunk>>huffmanValueShift][(f.b>>huffmanChunkBits)&h.linkMask]
n = uint(chunk & huffmanCountMask)
}
if n <= f.nb {
if n == 0 {
f.err = CorruptInputError(f.roffset)
return 0, f.err
}
f.b >>= n
f.nb -= n
return int(chunk >> huffmanValueShift), nil
}
}
}
func makeReader(r io.Reader) Reader {
if rr, ok := r.(Reader); ok {
return rr
}
return bufio.NewReader(r)
}
func fixedHuffmanDecoderInit() {
fixedOnce.Do(func() {
// These come from the RFC section 3.2.6.
var bits [288]int
for i := 0; i < 144; i++ {
bits[i] = 8
}
for i := 144; i < 256; i++ {
bits[i] = 9
}
for i := 256; i < 280; i++ {
bits[i] = 7
}
for i := 280; i < 288; i++ {
bits[i] = 8
}
fixedHuffmanDecoder.init(bits[:])
})
}
func (f *decompressor) Reset(r io.Reader, dict []byte) error {
*f = decompressor{
r: makeReader(r),
bits: f.bits,
codebits: f.codebits,
h1: f.h1,
h2: f.h2,
dict: f.dict,
step: (*decompressor).nextBlock,
}
f.dict.init(maxMatchOffset, dict)
return nil
}
// NewReader returns a new ReadCloser that can be used
// to read the uncompressed version of r.
// If r does not also implement io.ByteReader,
// the decompressor may read more data than necessary from r.
// It is the caller's responsibility to call Close on the ReadCloser
// when finished reading.
//
// The ReadCloser returned by NewReader also implements Resetter.
func NewReader(r io.Reader) io.ReadCloser {
fixedHuffmanDecoderInit()
var f decompressor
f.r = makeReader(r)
f.bits = new([maxNumLit + maxNumDist]int)
f.codebits = new([numCodes]int)
f.step = (*decompressor).nextBlock
f.dict.init(maxMatchOffset, nil)
return &f
}
// NewReaderDict is like NewReader but initializes the reader
// with a preset dictionary. The returned Reader behaves as if
// the uncompressed data stream started with the given dictionary,
// which has already been read. NewReaderDict is typically used
// to read data compressed by NewWriterDict.
//
// The ReadCloser returned by NewReader also implements Resetter.
func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser {
fixedHuffmanDecoderInit()
var f decompressor
f.r = makeReader(r)
f.bits = new([maxNumLit + maxNumDist]int)
f.codebits = new([numCodes]int)
f.step = (*decompressor).nextBlock
f.dict.init(maxMatchOffset, dict)
return &f
}
+48
View File
@@ -0,0 +1,48 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
var reverseByte = [256]byte{
0x00, 0x80, 0x40, 0xc0, 0x20, 0xa0, 0x60, 0xe0,
0x10, 0x90, 0x50, 0xd0, 0x30, 0xb0, 0x70, 0xf0,
0x08, 0x88, 0x48, 0xc8, 0x28, 0xa8, 0x68, 0xe8,
0x18, 0x98, 0x58, 0xd8, 0x38, 0xb8, 0x78, 0xf8,
0x04, 0x84, 0x44, 0xc4, 0x24, 0xa4, 0x64, 0xe4,
0x14, 0x94, 0x54, 0xd4, 0x34, 0xb4, 0x74, 0xf4,
0x0c, 0x8c, 0x4c, 0xcc, 0x2c, 0xac, 0x6c, 0xec,
0x1c, 0x9c, 0x5c, 0xdc, 0x3c, 0xbc, 0x7c, 0xfc,
0x02, 0x82, 0x42, 0xc2, 0x22, 0xa2, 0x62, 0xe2,
0x12, 0x92, 0x52, 0xd2, 0x32, 0xb2, 0x72, 0xf2,
0x0a, 0x8a, 0x4a, 0xca, 0x2a, 0xaa, 0x6a, 0xea,
0x1a, 0x9a, 0x5a, 0xda, 0x3a, 0xba, 0x7a, 0xfa,
0x06, 0x86, 0x46, 0xc6, 0x26, 0xa6, 0x66, 0xe6,
0x16, 0x96, 0x56, 0xd6, 0x36, 0xb6, 0x76, 0xf6,
0x0e, 0x8e, 0x4e, 0xce, 0x2e, 0xae, 0x6e, 0xee,
0x1e, 0x9e, 0x5e, 0xde, 0x3e, 0xbe, 0x7e, 0xfe,
0x01, 0x81, 0x41, 0xc1, 0x21, 0xa1, 0x61, 0xe1,
0x11, 0x91, 0x51, 0xd1, 0x31, 0xb1, 0x71, 0xf1,
0x09, 0x89, 0x49, 0xc9, 0x29, 0xa9, 0x69, 0xe9,
0x19, 0x99, 0x59, 0xd9, 0x39, 0xb9, 0x79, 0xf9,
0x05, 0x85, 0x45, 0xc5, 0x25, 0xa5, 0x65, 0xe5,
0x15, 0x95, 0x55, 0xd5, 0x35, 0xb5, 0x75, 0xf5,
0x0d, 0x8d, 0x4d, 0xcd, 0x2d, 0xad, 0x6d, 0xed,
0x1d, 0x9d, 0x5d, 0xdd, 0x3d, 0xbd, 0x7d, 0xfd,
0x03, 0x83, 0x43, 0xc3, 0x23, 0xa3, 0x63, 0xe3,
0x13, 0x93, 0x53, 0xd3, 0x33, 0xb3, 0x73, 0xf3,
0x0b, 0x8b, 0x4b, 0xcb, 0x2b, 0xab, 0x6b, 0xeb,
0x1b, 0x9b, 0x5b, 0xdb, 0x3b, 0xbb, 0x7b, 0xfb,
0x07, 0x87, 0x47, 0xc7, 0x27, 0xa7, 0x67, 0xe7,
0x17, 0x97, 0x57, 0xd7, 0x37, 0xb7, 0x77, 0xf7,
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef,
0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
}
func reverseUint16(v uint16) uint16 {
return uint16(reverseByte[v>>8]) | uint16(reverseByte[v&0xFF])<<8
}
func reverseBits(number uint16, bitLength byte) uint16 {
return reverseUint16(number << uint8(16-bitLength))
}
+900
View File
@@ -0,0 +1,900 @@
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Modified for deflate by Klaus Post (c) 2015.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// emitLiteral writes a literal chunk and returns the number of bytes written.
func emitLiteral(dst *tokens, lit []byte) {
ol := int(dst.n)
for i, v := range lit {
dst.tokens[(i+ol)&maxStoreBlockSize] = token(v)
}
dst.n += uint16(len(lit))
}
// emitCopy writes a copy chunk and returns the number of bytes written.
func emitCopy(dst *tokens, offset, length int) {
dst.tokens[dst.n] = matchToken(uint32(length-3), uint32(offset-minOffsetSize))
dst.n++
}
type snappyEnc interface {
Encode(dst *tokens, src []byte)
Reset()
}
func newSnappy(level int) snappyEnc {
switch level {
case 1:
return &snappyL1{}
case 2:
return &snappyL2{snappyGen: snappyGen{cur: maxStoreBlockSize, prev: make([]byte, 0, maxStoreBlockSize)}}
case 3:
return &snappyL3{snappyGen: snappyGen{cur: maxStoreBlockSize, prev: make([]byte, 0, maxStoreBlockSize)}}
case 4:
return &snappyL4{snappyL3{snappyGen: snappyGen{cur: maxStoreBlockSize, prev: make([]byte, 0, maxStoreBlockSize)}}}
default:
panic("invalid level specified")
}
}
const (
tableBits = 14 // Bits used in the table
tableSize = 1 << tableBits // Size of the table
tableMask = tableSize - 1 // Mask for table indices. Redundant, but can eliminate bounds checks.
tableShift = 32 - tableBits // Right-shift to get the tableBits most significant bits of a uint32.
baseMatchOffset = 1 // The smallest match offset
baseMatchLength = 3 // The smallest match length per the RFC section 3.2.5
maxMatchOffset = 1 << 15 // The largest match offset
)
func load32(b []byte, i int) uint32 {
b = b[i : i+4 : len(b)] // Help the compiler eliminate bounds checks on the next line.
return uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24
}
func load64(b []byte, i int) uint64 {
b = b[i : i+8 : len(b)] // Help the compiler eliminate bounds checks on the next line.
return uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 |
uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56
}
func hash(u uint32) uint32 {
return (u * 0x1e35a7bd) >> tableShift
}
// snappyL1 encapsulates level 1 compression
type snappyL1 struct{}
func (e *snappyL1) Reset() {}
func (e *snappyL1) Encode(dst *tokens, src []byte) {
const (
inputMargin = 16 - 1
minNonLiteralBlockSize = 1 + 1 + inputMargin
)
// This check isn't in the Snappy implementation, but there, the caller
// instead of the callee handles this case.
if len(src) < minNonLiteralBlockSize {
// We do not fill the token table.
// This will be picked up by caller.
dst.n = uint16(len(src))
return
}
// Initialize the hash table.
//
// The table element type is uint16, as s < sLimit and sLimit < len(src)
// and len(src) <= maxStoreBlockSize and maxStoreBlockSize == 65535.
var table [tableSize]uint16
// sLimit is when to stop looking for offset/length copies. The inputMargin
// lets us use a fast path for emitLiteral in the main loop, while we are
// looking for copies.
sLimit := len(src) - inputMargin
// nextEmit is where in src the next emitLiteral should start from.
nextEmit := 0
// The encoded form must start with a literal, as there are no previous
// bytes to copy, so we start looking for hash matches at s == 1.
s := 1
nextHash := hash(load32(src, s))
for {
// Copied from the C++ snappy implementation:
//
// Heuristic match skipping: If 32 bytes are scanned with no matches
// found, start looking only at every other byte. If 32 more bytes are
// scanned (or skipped), look at every third byte, etc.. When a match
// is found, immediately go back to looking at every byte. This is a
// small loss (~5% performance, ~0.1% density) for compressible data
// due to more bookkeeping, but for non-compressible data (such as
// JPEG) it's a huge win since the compressor quickly "realizes" the
// data is incompressible and doesn't bother looking for matches
// everywhere.
//
// The "skip" variable keeps track of how many bytes there are since
// the last match; dividing it by 32 (ie. right-shifting by five) gives
// the number of bytes to move ahead for each iteration.
skip := 32
nextS := s
candidate := 0
for {
s = nextS
bytesBetweenHashLookups := skip >> 5
nextS = s + bytesBetweenHashLookups
skip += bytesBetweenHashLookups
if nextS > sLimit {
goto emitRemainder
}
candidate = int(table[nextHash&tableMask])
table[nextHash&tableMask] = uint16(s)
nextHash = hash(load32(src, nextS))
if s-candidate <= maxMatchOffset && load32(src, s) == load32(src, candidate) {
break
}
}
// A 4-byte match has been found. We'll later see if more than 4 bytes
// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
// them as literal bytes.
emitLiteral(dst, src[nextEmit:s])
// Call emitCopy, and then see if another emitCopy could be our next
// move. Repeat until we find no match for the input immediately after
// what was consumed by the last emitCopy call.
//
// If we exit this loop normally then we need to call emitLiteral next,
// though we don't yet know how big the literal will be. We handle that
// by proceeding to the next iteration of the main loop. We also can
// exit this loop via goto if we get close to exhausting the input.
for {
// Invariant: we have a 4-byte match at s, and no need to emit any
// literal bytes prior to s.
base := s
// Extend the 4-byte match as long as possible.
//
// This is an inlined version of Snappy's:
// s = extendMatch(src, candidate+4, s+4)
s += 4
s1 := base + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
a := src[s:s1]
b := src[candidate+4:]
b = b[:len(a)]
l := len(a)
for i := range a {
if a[i] != b[i] {
l = i
break
}
}
s += l
// matchToken is flate's equivalent of Snappy's emitCopy.
dst.tokens[dst.n] = matchToken(uint32(s-base-baseMatchLength), uint32(base-candidate-baseMatchOffset))
dst.n++
nextEmit = s
if s >= sLimit {
goto emitRemainder
}
// We could immediately start working at s now, but to improve
// compression we first update the hash table at s-1 and at s. If
// another emitCopy is not our next move, also calculate nextHash
// at s+1. At least on GOARCH=amd64, these three hash calculations
// are faster as one load64 call (with some shifts) instead of
// three load32 calls.
x := load64(src, s-1)
prevHash := hash(uint32(x >> 0))
table[prevHash&tableMask] = uint16(s - 1)
currHash := hash(uint32(x >> 8))
candidate = int(table[currHash&tableMask])
table[currHash&tableMask] = uint16(s)
if s-candidate > maxMatchOffset || uint32(x>>8) != load32(src, candidate) {
nextHash = hash(uint32(x >> 16))
s++
break
}
}
}
emitRemainder:
if nextEmit < len(src) {
emitLiteral(dst, src[nextEmit:])
}
}
type tableEntry struct {
val uint32
offset int32
}
func load3232(b []byte, i int32) uint32 {
b = b[i : i+4 : len(b)] // Help the compiler eliminate bounds checks on the next line.
return uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24
}
func load6432(b []byte, i int32) uint64 {
b = b[i : i+8 : len(b)] // Help the compiler eliminate bounds checks on the next line.
return uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 |
uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56
}
// snappyGen maintains the table for matches,
// and the previous byte block for level 2.
// This is the generic implementation.
type snappyGen struct {
prev []byte
cur int32
}
// snappyGen maintains the table for matches,
// and the previous byte block for level 2.
// This is the generic implementation.
type snappyL2 struct {
snappyGen
table [tableSize]tableEntry
}
// EncodeL2 uses a similar algorithm to level 1, but is capable
// of matching across blocks giving better compression at a small slowdown.
func (e *snappyL2) Encode(dst *tokens, src []byte) {
const (
inputMargin = 8 - 1
minNonLiteralBlockSize = 1 + 1 + inputMargin
)
// Protect against e.cur wraparound.
if e.cur > 1<<30 {
for i := range e.table[:] {
e.table[i] = tableEntry{}
}
e.cur = maxStoreBlockSize
}
// This check isn't in the Snappy implementation, but there, the caller
// instead of the callee handles this case.
if len(src) < minNonLiteralBlockSize {
// We do not fill the token table.
// This will be picked up by caller.
dst.n = uint16(len(src))
e.cur += maxStoreBlockSize
e.prev = e.prev[:0]
return
}
// sLimit is when to stop looking for offset/length copies. The inputMargin
// lets us use a fast path for emitLiteral in the main loop, while we are
// looking for copies.
sLimit := int32(len(src) - inputMargin)
// nextEmit is where in src the next emitLiteral should start from.
nextEmit := int32(0)
s := int32(0)
cv := load3232(src, s)
nextHash := hash(cv)
for {
// Copied from the C++ snappy implementation:
//
// Heuristic match skipping: If 32 bytes are scanned with no matches
// found, start looking only at every other byte. If 32 more bytes are
// scanned (or skipped), look at every third byte, etc.. When a match
// is found, immediately go back to looking at every byte. This is a
// small loss (~5% performance, ~0.1% density) for compressible data
// due to more bookkeeping, but for non-compressible data (such as
// JPEG) it's a huge win since the compressor quickly "realizes" the
// data is incompressible and doesn't bother looking for matches
// everywhere.
//
// The "skip" variable keeps track of how many bytes there are since
// the last match; dividing it by 32 (ie. right-shifting by five) gives
// the number of bytes to move ahead for each iteration.
skip := int32(32)
nextS := s
var candidate tableEntry
for {
s = nextS
bytesBetweenHashLookups := skip >> 5
nextS = s + bytesBetweenHashLookups
skip += bytesBetweenHashLookups
if nextS > sLimit {
goto emitRemainder
}
candidate = e.table[nextHash&tableMask]
now := load3232(src, nextS)
e.table[nextHash&tableMask] = tableEntry{offset: s + e.cur, val: cv}
nextHash = hash(now)
offset := s - (candidate.offset - e.cur)
if offset > maxMatchOffset || cv != candidate.val {
// Out of range or not matched.
cv = now
continue
}
break
}
// A 4-byte match has been found. We'll later see if more than 4 bytes
// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
// them as literal bytes.
emitLiteral(dst, src[nextEmit:s])
// Call emitCopy, and then see if another emitCopy could be our next
// move. Repeat until we find no match for the input immediately after
// what was consumed by the last emitCopy call.
//
// If we exit this loop normally then we need to call emitLiteral next,
// though we don't yet know how big the literal will be. We handle that
// by proceeding to the next iteration of the main loop. We also can
// exit this loop via goto if we get close to exhausting the input.
for {
// Invariant: we have a 4-byte match at s, and no need to emit any
// literal bytes prior to s.
// Extend the 4-byte match as long as possible.
//
s += 4
t := candidate.offset - e.cur + 4
l := e.matchlen(s, t, src)
// matchToken is flate's equivalent of Snappy's emitCopy. (length,offset)
dst.tokens[dst.n] = matchToken(uint32(l+4-baseMatchLength), uint32(s-t-baseMatchOffset))
dst.n++
s += l
nextEmit = s
if s >= sLimit {
t += l
// Index first pair after match end.
if int(t+4) < len(src) && t > 0 {
cv := load3232(src, t)
e.table[hash(cv)&tableMask] = tableEntry{offset: t + e.cur, val: cv}
}
goto emitRemainder
}
// We could immediately start working at s now, but to improve
// compression we first update the hash table at s-1 and at s. If
// another emitCopy is not our next move, also calculate nextHash
// at s+1. At least on GOARCH=amd64, these three hash calculations
// are faster as one load64 call (with some shifts) instead of
// three load32 calls.
x := load6432(src, s-1)
prevHash := hash(uint32(x))
e.table[prevHash&tableMask] = tableEntry{offset: e.cur + s - 1, val: uint32(x)}
x >>= 8
currHash := hash(uint32(x))
candidate = e.table[currHash&tableMask]
e.table[currHash&tableMask] = tableEntry{offset: e.cur + s, val: uint32(x)}
offset := s - (candidate.offset - e.cur)
if offset > maxMatchOffset || uint32(x) != candidate.val {
cv = uint32(x >> 8)
nextHash = hash(cv)
s++
break
}
}
}
emitRemainder:
if int(nextEmit) < len(src) {
emitLiteral(dst, src[nextEmit:])
}
e.cur += int32(len(src))
e.prev = e.prev[:len(src)]
copy(e.prev, src)
}
type tableEntryPrev struct {
Cur tableEntry
Prev tableEntry
}
// snappyL3
type snappyL3 struct {
snappyGen
table [tableSize]tableEntryPrev
}
// Encode uses a similar algorithm to level 2, will check up to two candidates.
func (e *snappyL3) Encode(dst *tokens, src []byte) {
const (
inputMargin = 8 - 1
minNonLiteralBlockSize = 1 + 1 + inputMargin
)
// Protect against e.cur wraparound.
if e.cur > 1<<30 {
for i := range e.table[:] {
e.table[i] = tableEntryPrev{}
}
e.snappyGen = snappyGen{cur: maxStoreBlockSize, prev: e.prev[:0]}
}
// This check isn't in the Snappy implementation, but there, the caller
// instead of the callee handles this case.
if len(src) < minNonLiteralBlockSize {
// We do not fill the token table.
// This will be picked up by caller.
dst.n = uint16(len(src))
e.cur += maxStoreBlockSize
e.prev = e.prev[:0]
return
}
// sLimit is when to stop looking for offset/length copies. The inputMargin
// lets us use a fast path for emitLiteral in the main loop, while we are
// looking for copies.
sLimit := int32(len(src) - inputMargin)
// nextEmit is where in src the next emitLiteral should start from.
nextEmit := int32(0)
s := int32(0)
cv := load3232(src, s)
nextHash := hash(cv)
for {
// Copied from the C++ snappy implementation:
//
// Heuristic match skipping: If 32 bytes are scanned with no matches
// found, start looking only at every other byte. If 32 more bytes are
// scanned (or skipped), look at every third byte, etc.. When a match
// is found, immediately go back to looking at every byte. This is a
// small loss (~5% performance, ~0.1% density) for compressible data
// due to more bookkeeping, but for non-compressible data (such as
// JPEG) it's a huge win since the compressor quickly "realizes" the
// data is incompressible and doesn't bother looking for matches
// everywhere.
//
// The "skip" variable keeps track of how many bytes there are since
// the last match; dividing it by 32 (ie. right-shifting by five) gives
// the number of bytes to move ahead for each iteration.
skip := int32(32)
nextS := s
var candidate tableEntry
for {
s = nextS
bytesBetweenHashLookups := skip >> 5
nextS = s + bytesBetweenHashLookups
skip += bytesBetweenHashLookups
if nextS > sLimit {
goto emitRemainder
}
candidates := e.table[nextHash&tableMask]
now := load3232(src, nextS)
e.table[nextHash&tableMask] = tableEntryPrev{Prev: candidates.Cur, Cur: tableEntry{offset: s + e.cur, val: cv}}
nextHash = hash(now)
// Check both candidates
candidate = candidates.Cur
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
break
}
} else {
// We only check if value mismatches.
// Offset will always be invalid in other cases.
candidate = candidates.Prev
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
break
}
}
}
cv = now
}
// A 4-byte match has been found. We'll later see if more than 4 bytes
// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
// them as literal bytes.
emitLiteral(dst, src[nextEmit:s])
// Call emitCopy, and then see if another emitCopy could be our next
// move. Repeat until we find no match for the input immediately after
// what was consumed by the last emitCopy call.
//
// If we exit this loop normally then we need to call emitLiteral next,
// though we don't yet know how big the literal will be. We handle that
// by proceeding to the next iteration of the main loop. We also can
// exit this loop via goto if we get close to exhausting the input.
for {
// Invariant: we have a 4-byte match at s, and no need to emit any
// literal bytes prior to s.
// Extend the 4-byte match as long as possible.
//
s += 4
t := candidate.offset - e.cur + 4
l := e.matchlen(s, t, src)
// matchToken is flate's equivalent of Snappy's emitCopy. (length,offset)
dst.tokens[dst.n] = matchToken(uint32(l+4-baseMatchLength), uint32(s-t-baseMatchOffset))
dst.n++
s += l
nextEmit = s
if s >= sLimit {
t += l
// Index first pair after match end.
if int(t+4) < len(src) && t > 0 {
cv := load3232(src, t)
nextHash = hash(cv)
e.table[nextHash&tableMask] = tableEntryPrev{
Prev: e.table[nextHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + t, val: cv},
}
}
goto emitRemainder
}
// We could immediately start working at s now, but to improve
// compression we first update the hash table at s-3 to s. If
// another emitCopy is not our next move, also calculate nextHash
// at s+1. At least on GOARCH=amd64, these three hash calculations
// are faster as one load64 call (with some shifts) instead of
// three load32 calls.
x := load6432(src, s-3)
prevHash := hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 3, val: uint32(x)},
}
x >>= 8
prevHash = hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 2, val: uint32(x)},
}
x >>= 8
prevHash = hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 1, val: uint32(x)},
}
x >>= 8
currHash := hash(uint32(x))
candidates := e.table[currHash&tableMask]
cv = uint32(x)
e.table[currHash&tableMask] = tableEntryPrev{
Prev: candidates.Cur,
Cur: tableEntry{offset: s + e.cur, val: cv},
}
// Check both candidates
candidate = candidates.Cur
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
continue
}
} else {
// We only check if value mismatches.
// Offset will always be invalid in other cases.
candidate = candidates.Prev
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
continue
}
}
}
cv = uint32(x >> 8)
nextHash = hash(cv)
s++
break
}
}
emitRemainder:
if int(nextEmit) < len(src) {
emitLiteral(dst, src[nextEmit:])
}
e.cur += int32(len(src))
e.prev = e.prev[:len(src)]
copy(e.prev, src)
}
// snappyL4
type snappyL4 struct {
snappyL3
}
// Encode uses a similar algorithm to level 3,
// but will check up to two candidates if first isn't long enough.
func (e *snappyL4) Encode(dst *tokens, src []byte) {
const (
inputMargin = 8 - 3
minNonLiteralBlockSize = 1 + 1 + inputMargin
matchLenGood = 12
)
// Protect against e.cur wraparound.
if e.cur > 1<<30 {
for i := range e.table[:] {
e.table[i] = tableEntryPrev{}
}
e.snappyGen = snappyGen{cur: maxStoreBlockSize, prev: e.prev[:0]}
}
// This check isn't in the Snappy implementation, but there, the caller
// instead of the callee handles this case.
if len(src) < minNonLiteralBlockSize {
// We do not fill the token table.
// This will be picked up by caller.
dst.n = uint16(len(src))
e.cur += maxStoreBlockSize
e.prev = e.prev[:0]
return
}
// sLimit is when to stop looking for offset/length copies. The inputMargin
// lets us use a fast path for emitLiteral in the main loop, while we are
// looking for copies.
sLimit := int32(len(src) - inputMargin)
// nextEmit is where in src the next emitLiteral should start from.
nextEmit := int32(0)
s := int32(0)
cv := load3232(src, s)
nextHash := hash(cv)
for {
// Copied from the C++ snappy implementation:
//
// Heuristic match skipping: If 32 bytes are scanned with no matches
// found, start looking only at every other byte. If 32 more bytes are
// scanned (or skipped), look at every third byte, etc.. When a match
// is found, immediately go back to looking at every byte. This is a
// small loss (~5% performance, ~0.1% density) for compressible data
// due to more bookkeeping, but for non-compressible data (such as
// JPEG) it's a huge win since the compressor quickly "realizes" the
// data is incompressible and doesn't bother looking for matches
// everywhere.
//
// The "skip" variable keeps track of how many bytes there are since
// the last match; dividing it by 32 (ie. right-shifting by five) gives
// the number of bytes to move ahead for each iteration.
skip := int32(32)
nextS := s
var candidate tableEntry
var candidateAlt tableEntry
for {
s = nextS
bytesBetweenHashLookups := skip >> 5
nextS = s + bytesBetweenHashLookups
skip += bytesBetweenHashLookups
if nextS > sLimit {
goto emitRemainder
}
candidates := e.table[nextHash&tableMask]
now := load3232(src, nextS)
e.table[nextHash&tableMask] = tableEntryPrev{Prev: candidates.Cur, Cur: tableEntry{offset: s + e.cur, val: cv}}
nextHash = hash(now)
// Check both candidates
candidate = candidates.Cur
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset < maxMatchOffset {
offset = s - (candidates.Prev.offset - e.cur)
if cv == candidates.Prev.val && offset < maxMatchOffset {
candidateAlt = candidates.Prev
}
break
}
} else {
// We only check if value mismatches.
// Offset will always be invalid in other cases.
candidate = candidates.Prev
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset < maxMatchOffset {
break
}
}
}
cv = now
}
// A 4-byte match has been found. We'll later see if more than 4 bytes
// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
// them as literal bytes.
emitLiteral(dst, src[nextEmit:s])
// Call emitCopy, and then see if another emitCopy could be our next
// move. Repeat until we find no match for the input immediately after
// what was consumed by the last emitCopy call.
//
// If we exit this loop normally then we need to call emitLiteral next,
// though we don't yet know how big the literal will be. We handle that
// by proceeding to the next iteration of the main loop. We also can
// exit this loop via goto if we get close to exhausting the input.
for {
// Invariant: we have a 4-byte match at s, and no need to emit any
// literal bytes prior to s.
// Extend the 4-byte match as long as possible.
//
s += 4
t := candidate.offset - e.cur + 4
l := e.matchlen(s, t, src)
// Try alternative candidate if match length < matchLenGood.
if l < matchLenGood-4 && candidateAlt.offset != 0 {
t2 := candidateAlt.offset - e.cur + 4
l2 := e.matchlen(s, t2, src)
if l2 > l {
l = l2
t = t2
}
}
// matchToken is flate's equivalent of Snappy's emitCopy. (length,offset)
dst.tokens[dst.n] = matchToken(uint32(l+4-baseMatchLength), uint32(s-t-baseMatchOffset))
dst.n++
s += l
nextEmit = s
if s >= sLimit {
t += l
// Index first pair after match end.
if int(t+4) < len(src) && t > 0 {
cv := load3232(src, t)
nextHash = hash(cv)
e.table[nextHash&tableMask] = tableEntryPrev{
Prev: e.table[nextHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + t, val: cv},
}
}
goto emitRemainder
}
// We could immediately start working at s now, but to improve
// compression we first update the hash table at s-3 to s. If
// another emitCopy is not our next move, also calculate nextHash
// at s+1. At least on GOARCH=amd64, these three hash calculations
// are faster as one load64 call (with some shifts) instead of
// three load32 calls.
x := load6432(src, s-3)
prevHash := hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 3, val: uint32(x)},
}
x >>= 8
prevHash = hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 2, val: uint32(x)},
}
x >>= 8
prevHash = hash(uint32(x))
e.table[prevHash&tableMask] = tableEntryPrev{
Prev: e.table[prevHash&tableMask].Cur,
Cur: tableEntry{offset: e.cur + s - 1, val: uint32(x)},
}
x >>= 8
currHash := hash(uint32(x))
candidates := e.table[currHash&tableMask]
cv = uint32(x)
e.table[currHash&tableMask] = tableEntryPrev{
Prev: candidates.Cur,
Cur: tableEntry{offset: s + e.cur, val: cv},
}
// Check both candidates
candidate = candidates.Cur
candidateAlt = tableEntry{}
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
offset = s - (candidates.Prev.offset - e.cur)
if cv == candidates.Prev.val && offset <= maxMatchOffset {
candidateAlt = candidates.Prev
}
continue
}
} else {
// We only check if value mismatches.
// Offset will always be invalid in other cases.
candidate = candidates.Prev
if cv == candidate.val {
offset := s - (candidate.offset - e.cur)
if offset <= maxMatchOffset {
continue
}
}
}
cv = uint32(x >> 8)
nextHash = hash(cv)
s++
break
}
}
emitRemainder:
if int(nextEmit) < len(src) {
emitLiteral(dst, src[nextEmit:])
}
e.cur += int32(len(src))
e.prev = e.prev[:len(src)]
copy(e.prev, src)
}
func (e *snappyGen) matchlen(s, t int32, src []byte) int32 {
s1 := int(s) + maxMatchLength - 4
if s1 > len(src) {
s1 = len(src)
}
// If we are inside the current block
if t >= 0 {
b := src[t:]
a := src[s:s1]
b = b[:len(a)]
// Extend the match to be as long as possible.
for i := range a {
if a[i] != b[i] {
return int32(i)
}
}
return int32(len(a))
}
// We found a match in the previous block.
tp := int32(len(e.prev)) + t
if tp < 0 {
return 0
}
// Extend the match to be as long as possible.
a := src[s:s1]
b := e.prev[tp:]
if len(b) > len(a) {
b = b[:len(a)]
}
a = a[:len(b)]
for i := range b {
if a[i] != b[i] {
return int32(i)
}
}
// If we reached our limit, we matched everything we are
// allowed to in the previous block and we return.
n := int32(len(b))
if int(s+n) == s1 {
return n
}
// Continue looking for more matches in the current block.
a = src[s+n : s1]
b = src[:len(a)]
for i := range a {
if a[i] != b[i] {
return int32(i) + n
}
}
return int32(len(a)) + n
}
// Reset the encoding table.
func (e *snappyGen) Reset() {
e.prev = e.prev[:0]
e.cur += maxMatchOffset
}
+115
View File
@@ -0,0 +1,115 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import "fmt"
const (
// 2 bits: type 0 = literal 1=EOF 2=Match 3=Unused
// 8 bits: xlength = length - MIN_MATCH_LENGTH
// 22 bits xoffset = offset - MIN_OFFSET_SIZE, or literal
lengthShift = 22
offsetMask = 1<<lengthShift - 1
typeMask = 3 << 30
literalType = 0 << 30
matchType = 1 << 30
)
// The length code for length X (MIN_MATCH_LENGTH <= X <= MAX_MATCH_LENGTH)
// is lengthCodes[length - MIN_MATCH_LENGTH]
var lengthCodes = [...]uint32{
0, 1, 2, 3, 4, 5, 6, 7, 8, 8,
9, 9, 10, 10, 11, 11, 12, 12, 12, 12,
13, 13, 13, 13, 14, 14, 14, 14, 15, 15,
15, 15, 16, 16, 16, 16, 16, 16, 16, 16,
17, 17, 17, 17, 17, 17, 17, 17, 18, 18,
18, 18, 18, 18, 18, 18, 19, 19, 19, 19,
19, 19, 19, 19, 20, 20, 20, 20, 20, 20,
20, 20, 20, 20, 20, 20, 20, 20, 20, 20,
21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
21, 21, 21, 21, 21, 21, 22, 22, 22, 22,
22, 22, 22, 22, 22, 22, 22, 22, 22, 22,
22, 22, 23, 23, 23, 23, 23, 23, 23, 23,
23, 23, 23, 23, 23, 23, 23, 23, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 28,
}
var offsetCodes = [...]uint32{
0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7,
8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
}
type token uint32
type tokens struct {
tokens [maxStoreBlockSize + 1]token
n uint16 // Must be able to contain maxStoreBlockSize
}
// Convert a literal into a literal token.
func literalToken(literal uint32) token { return token(literalType + literal) }
// Convert a < xlength, xoffset > pair into a match token.
func matchToken(xlength uint32, xoffset uint32) token {
return token(matchType + xlength<<lengthShift + xoffset)
}
func matchTokend(xlength uint32, xoffset uint32) token {
if xlength > maxMatchLength || xoffset > maxMatchOffset {
panic(fmt.Sprintf("Invalid match: len: %d, offset: %d\n", xlength, xoffset))
return token(matchType)
}
return token(matchType + xlength<<lengthShift + xoffset)
}
// Returns the type of a token
func (t token) typ() uint32 { return uint32(t) & typeMask }
// Returns the literal of a literal token
func (t token) literal() uint32 { return uint32(t - literalType) }
// Returns the extra offset of a match token
func (t token) offset() uint32 { return uint32(t) & offsetMask }
func (t token) length() uint32 { return uint32((t - matchType) >> lengthShift) }
func lengthCode(len uint32) uint32 { return lengthCodes[len] }
// Returns the offset code corresponding to a specific offset
func offsetCode(off uint32) uint32 {
if off < uint32(len(offsetCodes)) {
return offsetCodes[off]
} else if off>>7 < uint32(len(offsetCodes)) {
return offsetCodes[off>>7] + 14
} else {
return offsetCodes[off>>14] + 28
}
}
+344
View File
@@ -0,0 +1,344 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package gzip implements reading and writing of gzip format compressed files,
// as specified in RFC 1952.
package gzip
import (
"bufio"
"encoding/binary"
"errors"
"hash/crc32"
"io"
"time"
"github.com/klauspost/compress/flate"
)
const (
gzipID1 = 0x1f
gzipID2 = 0x8b
gzipDeflate = 8
flagText = 1 << 0
flagHdrCrc = 1 << 1
flagExtra = 1 << 2
flagName = 1 << 3
flagComment = 1 << 4
)
var (
// ErrChecksum is returned when reading GZIP data that has an invalid checksum.
ErrChecksum = errors.New("gzip: invalid checksum")
// ErrHeader is returned when reading GZIP data that has an invalid header.
ErrHeader = errors.New("gzip: invalid header")
)
var le = binary.LittleEndian
// noEOF converts io.EOF to io.ErrUnexpectedEOF.
func noEOF(err error) error {
if err == io.EOF {
return io.ErrUnexpectedEOF
}
return err
}
// The gzip file stores a header giving metadata about the compressed file.
// That header is exposed as the fields of the Writer and Reader structs.
//
// Strings must be UTF-8 encoded and may only contain Unicode code points
// U+0001 through U+00FF, due to limitations of the GZIP file format.
type Header struct {
Comment string // comment
Extra []byte // "extra data"
ModTime time.Time // modification time
Name string // file name
OS byte // operating system type
}
// A Reader is an io.Reader that can be read to retrieve
// uncompressed data from a gzip-format compressed file.
//
// In general, a gzip file can be a concatenation of gzip files,
// each with its own header. Reads from the Reader
// return the concatenation of the uncompressed data of each.
// Only the first header is recorded in the Reader fields.
//
// Gzip files store a length and checksum of the uncompressed data.
// The Reader will return a ErrChecksum when Read
// reaches the end of the uncompressed data if it does not
// have the expected length or checksum. Clients should treat data
// returned by Read as tentative until they receive the io.EOF
// marking the end of the data.
type Reader struct {
Header // valid after NewReader or Reader.Reset
r flate.Reader
decompressor io.ReadCloser
digest uint32 // CRC-32, IEEE polynomial (section 8)
size uint32 // Uncompressed size (section 2.3.1)
buf [512]byte
err error
multistream bool
}
// NewReader creates a new Reader reading the given reader.
// If r does not also implement io.ByteReader,
// the decompressor may read more data than necessary from r.
//
// It is the caller's responsibility to call Close on the Reader when done.
//
// The Reader.Header fields will be valid in the Reader returned.
func NewReader(r io.Reader) (*Reader, error) {
z := new(Reader)
if err := z.Reset(r); err != nil {
return nil, err
}
return z, nil
}
// Reset discards the Reader z's state and makes it equivalent to the
// result of its original state from NewReader, but reading from r instead.
// This permits reusing a Reader rather than allocating a new one.
func (z *Reader) Reset(r io.Reader) error {
*z = Reader{
decompressor: z.decompressor,
multistream: true,
}
if rr, ok := r.(flate.Reader); ok {
z.r = rr
} else {
z.r = bufio.NewReader(r)
}
z.Header, z.err = z.readHeader()
return z.err
}
// Multistream controls whether the reader supports multistream files.
//
// If enabled (the default), the Reader expects the input to be a sequence
// of individually gzipped data streams, each with its own header and
// trailer, ending at EOF. The effect is that the concatenation of a sequence
// of gzipped files is treated as equivalent to the gzip of the concatenation
// of the sequence. This is standard behavior for gzip readers.
//
// Calling Multistream(false) disables this behavior; disabling the behavior
// can be useful when reading file formats that distinguish individual gzip
// data streams or mix gzip data streams with other data streams.
// In this mode, when the Reader reaches the end of the data stream,
// Read returns io.EOF. If the underlying reader implements io.ByteReader,
// it will be left positioned just after the gzip stream.
// To start the next stream, call z.Reset(r) followed by z.Multistream(false).
// If there is no next stream, z.Reset(r) will return io.EOF.
func (z *Reader) Multistream(ok bool) {
z.multistream = ok
}
// readString reads a NUL-terminated string from z.r.
// It treats the bytes read as being encoded as ISO 8859-1 (Latin-1) and
// will output a string encoded using UTF-8.
// This method always updates z.digest with the data read.
func (z *Reader) readString() (string, error) {
var err error
needConv := false
for i := 0; ; i++ {
if i >= len(z.buf) {
return "", ErrHeader
}
z.buf[i], err = z.r.ReadByte()
if err != nil {
return "", err
}
if z.buf[i] > 0x7f {
needConv = true
}
if z.buf[i] == 0 {
// Digest covers the NUL terminator.
z.digest = crc32.Update(z.digest, crc32.IEEETable, z.buf[:i+1])
// Strings are ISO 8859-1, Latin-1 (RFC 1952, section 2.3.1).
if needConv {
s := make([]rune, 0, i)
for _, v := range z.buf[:i] {
s = append(s, rune(v))
}
return string(s), nil
}
return string(z.buf[:i]), nil
}
}
}
// readHeader reads the GZIP header according to section 2.3.1.
// This method does not set z.err.
func (z *Reader) readHeader() (hdr Header, err error) {
if _, err = io.ReadFull(z.r, z.buf[:10]); err != nil {
// RFC 1952, section 2.2, says the following:
// A gzip file consists of a series of "members" (compressed data sets).
//
// Other than this, the specification does not clarify whether a
// "series" is defined as "one or more" or "zero or more". To err on the
// side of caution, Go interprets this to mean "zero or more".
// Thus, it is okay to return io.EOF here.
return hdr, err
}
if z.buf[0] != gzipID1 || z.buf[1] != gzipID2 || z.buf[2] != gzipDeflate {
return hdr, ErrHeader
}
flg := z.buf[3]
hdr.ModTime = time.Unix(int64(le.Uint32(z.buf[4:8])), 0)
// z.buf[8] is XFL and is currently ignored.
hdr.OS = z.buf[9]
z.digest = crc32.ChecksumIEEE(z.buf[:10])
if flg&flagExtra != 0 {
if _, err = io.ReadFull(z.r, z.buf[:2]); err != nil {
return hdr, noEOF(err)
}
z.digest = crc32.Update(z.digest, crc32.IEEETable, z.buf[:2])
data := make([]byte, le.Uint16(z.buf[:2]))
if _, err = io.ReadFull(z.r, data); err != nil {
return hdr, noEOF(err)
}
z.digest = crc32.Update(z.digest, crc32.IEEETable, data)
hdr.Extra = data
}
var s string
if flg&flagName != 0 {
if s, err = z.readString(); err != nil {
return hdr, err
}
hdr.Name = s
}
if flg&flagComment != 0 {
if s, err = z.readString(); err != nil {
return hdr, err
}
hdr.Comment = s
}
if flg&flagHdrCrc != 0 {
if _, err = io.ReadFull(z.r, z.buf[:2]); err != nil {
return hdr, noEOF(err)
}
digest := le.Uint16(z.buf[:2])
if digest != uint16(z.digest) {
return hdr, ErrHeader
}
}
z.digest = 0
if z.decompressor == nil {
z.decompressor = flate.NewReader(z.r)
} else {
z.decompressor.(flate.Resetter).Reset(z.r, nil)
}
return hdr, nil
}
// Read implements io.Reader, reading uncompressed bytes from its underlying Reader.
func (z *Reader) Read(p []byte) (n int, err error) {
if z.err != nil {
return 0, z.err
}
n, z.err = z.decompressor.Read(p)
z.digest = crc32.Update(z.digest, crc32.IEEETable, p[:n])
z.size += uint32(n)
if z.err != io.EOF {
// In the normal case we return here.
return n, z.err
}
// Finished file; check checksum and size.
if _, err := io.ReadFull(z.r, z.buf[:8]); err != nil {
z.err = noEOF(err)
return n, z.err
}
digest := le.Uint32(z.buf[:4])
size := le.Uint32(z.buf[4:8])
if digest != z.digest || size != z.size {
z.err = ErrChecksum
return n, z.err
}
z.digest, z.size = 0, 0
// File is ok; check if there is another.
if !z.multistream {
return n, io.EOF
}
z.err = nil // Remove io.EOF
if _, z.err = z.readHeader(); z.err != nil {
return n, z.err
}
// Read from next file, if necessary.
if n > 0 {
return n, nil
}
return z.Read(p)
}
// Support the io.WriteTo interface for io.Copy and friends.
func (z *Reader) WriteTo(w io.Writer) (int64, error) {
total := int64(0)
crcWriter := crc32.NewIEEE()
for {
if z.err != nil {
if z.err == io.EOF {
return total, nil
}
return total, z.err
}
// We write both to output and digest.
mw := io.MultiWriter(w, crcWriter)
n, err := z.decompressor.(io.WriterTo).WriteTo(mw)
total += n
z.size += uint32(n)
if err != nil {
z.err = err
return total, z.err
}
// Finished file; check checksum + size.
if _, err := io.ReadFull(z.r, z.buf[0:8]); err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
z.err = err
return total, err
}
z.digest = crcWriter.Sum32()
digest := le.Uint32(z.buf[:4])
size := le.Uint32(z.buf[4:8])
if digest != z.digest || size != z.size {
z.err = ErrChecksum
return total, z.err
}
z.digest, z.size = 0, 0
// File is ok; check if there is another.
if !z.multistream {
return total, nil
}
crcWriter.Reset()
z.err = nil // Remove io.EOF
if _, z.err = z.readHeader(); z.err != nil {
if z.err == io.EOF {
return total, nil
}
return total, z.err
}
}
}
// Close closes the Reader. It does not close the underlying io.Reader.
// In order for the GZIP checksum to be verified, the reader must be
// fully consumed until the io.EOF.
func (z *Reader) Close() error { return z.decompressor.Close() }
+251
View File
@@ -0,0 +1,251 @@
// Copyright 2010 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package gzip
import (
"errors"
"fmt"
"hash/crc32"
"io"
"github.com/klauspost/compress/flate"
)
// These constants are copied from the flate package, so that code that imports
// "compress/gzip" does not also have to import "compress/flate".
const (
NoCompression = flate.NoCompression
BestSpeed = flate.BestSpeed
BestCompression = flate.BestCompression
DefaultCompression = flate.DefaultCompression
ConstantCompression = flate.ConstantCompression
HuffmanOnly = flate.HuffmanOnly
)
// A Writer is an io.WriteCloser.
// Writes to a Writer are compressed and written to w.
type Writer struct {
Header // written at first call to Write, Flush, or Close
w io.Writer
level int
wroteHeader bool
compressor *flate.Writer
digest uint32 // CRC-32, IEEE polynomial (section 8)
size uint32 // Uncompressed size (section 2.3.1)
closed bool
buf [10]byte
err error
}
// NewWriter returns a new Writer.
// Writes to the returned writer are compressed and written to w.
//
// It is the caller's responsibility to call Close on the WriteCloser when done.
// Writes may be buffered and not flushed until Close.
//
// Callers that wish to set the fields in Writer.Header must do so before
// the first call to Write, Flush, or Close.
func NewWriter(w io.Writer) *Writer {
z, _ := NewWriterLevel(w, DefaultCompression)
return z
}
// NewWriterLevel is like NewWriter but specifies the compression level instead
// of assuming DefaultCompression.
//
// The compression level can be DefaultCompression, NoCompression, or any
// integer value between BestSpeed and BestCompression inclusive. The error
// returned will be nil if the level is valid.
func NewWriterLevel(w io.Writer, level int) (*Writer, error) {
if level < HuffmanOnly || level > BestCompression {
return nil, fmt.Errorf("gzip: invalid compression level: %d", level)
}
z := new(Writer)
z.init(w, level)
return z, nil
}
func (z *Writer) init(w io.Writer, level int) {
compressor := z.compressor
if compressor != nil {
compressor.Reset(w)
}
*z = Writer{
Header: Header{
OS: 255, // unknown
},
w: w,
level: level,
compressor: compressor,
}
}
// Reset discards the Writer z's state and makes it equivalent to the
// result of its original state from NewWriter or NewWriterLevel, but
// writing to w instead. This permits reusing a Writer rather than
// allocating a new one.
func (z *Writer) Reset(w io.Writer) {
z.init(w, z.level)
}
// writeBytes writes a length-prefixed byte slice to z.w.
func (z *Writer) writeBytes(b []byte) error {
if len(b) > 0xffff {
return errors.New("gzip.Write: Extra data is too large")
}
le.PutUint16(z.buf[:2], uint16(len(b)))
_, err := z.w.Write(z.buf[:2])
if err != nil {
return err
}
_, err = z.w.Write(b)
return err
}
// writeString writes a UTF-8 string s in GZIP's format to z.w.
// GZIP (RFC 1952) specifies that strings are NUL-terminated ISO 8859-1 (Latin-1).
func (z *Writer) writeString(s string) (err error) {
// GZIP stores Latin-1 strings; error if non-Latin-1; convert if non-ASCII.
needconv := false
for _, v := range s {
if v == 0 || v > 0xff {
return errors.New("gzip.Write: non-Latin-1 header string")
}
if v > 0x7f {
needconv = true
}
}
if needconv {
b := make([]byte, 0, len(s))
for _, v := range s {
b = append(b, byte(v))
}
_, err = z.w.Write(b)
} else {
_, err = io.WriteString(z.w, s)
}
if err != nil {
return err
}
// GZIP strings are NUL-terminated.
z.buf[0] = 0
_, err = z.w.Write(z.buf[:1])
return err
}
// Write writes a compressed form of p to the underlying io.Writer. The
// compressed bytes are not necessarily flushed until the Writer is closed.
func (z *Writer) Write(p []byte) (int, error) {
if z.err != nil {
return 0, z.err
}
var n int
// Write the GZIP header lazily.
if !z.wroteHeader {
z.wroteHeader = true
z.buf[0] = gzipID1
z.buf[1] = gzipID2
z.buf[2] = gzipDeflate
z.buf[3] = 0
if z.Extra != nil {
z.buf[3] |= 0x04
}
if z.Name != "" {
z.buf[3] |= 0x08
}
if z.Comment != "" {
z.buf[3] |= 0x10
}
le.PutUint32(z.buf[4:8], uint32(z.ModTime.Unix()))
if z.level == BestCompression {
z.buf[8] = 2
} else if z.level == BestSpeed {
z.buf[8] = 4
} else {
z.buf[8] = 0
}
z.buf[9] = z.OS
n, z.err = z.w.Write(z.buf[:10])
if z.err != nil {
return n, z.err
}
if z.Extra != nil {
z.err = z.writeBytes(z.Extra)
if z.err != nil {
return n, z.err
}
}
if z.Name != "" {
z.err = z.writeString(z.Name)
if z.err != nil {
return n, z.err
}
}
if z.Comment != "" {
z.err = z.writeString(z.Comment)
if z.err != nil {
return n, z.err
}
}
if z.compressor == nil {
z.compressor, _ = flate.NewWriter(z.w, z.level)
}
}
z.size += uint32(len(p))
z.digest = crc32.Update(z.digest, crc32.IEEETable, p)
n, z.err = z.compressor.Write(p)
return n, z.err
}
// Flush flushes any pending compressed data to the underlying writer.
//
// It is useful mainly in compressed network protocols, to ensure that
// a remote reader has enough data to reconstruct a packet. Flush does
// not return until the data has been written. If the underlying
// writer returns an error, Flush returns that error.
//
// In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.
func (z *Writer) Flush() error {
if z.err != nil {
return z.err
}
if z.closed {
return nil
}
if !z.wroteHeader {
z.Write(nil)
if z.err != nil {
return z.err
}
}
z.err = z.compressor.Flush()
return z.err
}
// Close closes the Writer, flushing any unwritten data to the underlying
// io.Writer, but does not close the underlying io.Writer.
func (z *Writer) Close() error {
if z.err != nil {
return z.err
}
if z.closed {
return nil
}
z.closed = true
if !z.wroteHeader {
z.Write(nil)
if z.err != nil {
return z.err
}
}
z.err = z.compressor.Close()
if z.err != nil {
return z.err
}
le.PutUint32(z.buf[:4], z.digest)
le.PutUint32(z.buf[4:8], z.size)
_, z.err = z.w.Write(z.buf[:8])
return z.err
}
+183
View File
@@ -0,0 +1,183 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
/*
Package zlib implements reading and writing of zlib format compressed data,
as specified in RFC 1950.
The implementation provides filters that uncompress during reading
and compress during writing. For example, to write compressed data
to a buffer:
var b bytes.Buffer
w := zlib.NewWriter(&b)
w.Write([]byte("hello, world\n"))
w.Close()
and to read that data back:
r, err := zlib.NewReader(&b)
io.Copy(os.Stdout, r)
r.Close()
*/
package zlib
import (
"bufio"
"errors"
"hash"
"hash/adler32"
"io"
"github.com/klauspost/compress/flate"
)
const zlibDeflate = 8
var (
// ErrChecksum is returned when reading ZLIB data that has an invalid checksum.
ErrChecksum = errors.New("zlib: invalid checksum")
// ErrDictionary is returned when reading ZLIB data that has an invalid dictionary.
ErrDictionary = errors.New("zlib: invalid dictionary")
// ErrHeader is returned when reading ZLIB data that has an invalid header.
ErrHeader = errors.New("zlib: invalid header")
)
type reader struct {
r flate.Reader
decompressor io.ReadCloser
digest hash.Hash32
err error
scratch [4]byte
}
// Resetter resets a ReadCloser returned by NewReader or NewReaderDict to
// to switch to a new underlying Reader. This permits reusing a ReadCloser
// instead of allocating a new one.
type Resetter interface {
// Reset discards any buffered data and resets the Resetter as if it was
// newly initialized with the given reader.
Reset(r io.Reader, dict []byte) error
}
// NewReader creates a new ReadCloser.
// Reads from the returned ReadCloser read and decompress data from r.
// If r does not implement io.ByteReader, the decompressor may read more
// data than necessary from r.
// It is the caller's responsibility to call Close on the ReadCloser when done.
//
// The ReadCloser returned by NewReader also implements Resetter.
func NewReader(r io.Reader) (io.ReadCloser, error) {
return NewReaderDict(r, nil)
}
// NewReaderDict is like NewReader but uses a preset dictionary.
// NewReaderDict ignores the dictionary if the compressed data does not refer to it.
// If the compressed data refers to a different dictionary, NewReaderDict returns ErrDictionary.
//
// The ReadCloser returned by NewReaderDict also implements Resetter.
func NewReaderDict(r io.Reader, dict []byte) (io.ReadCloser, error) {
z := new(reader)
err := z.Reset(r, dict)
if err != nil {
return nil, err
}
return z, nil
}
func (z *reader) Read(p []byte) (int, error) {
if z.err != nil {
return 0, z.err
}
var n int
n, z.err = z.decompressor.Read(p)
z.digest.Write(p[0:n])
if z.err != io.EOF {
// In the normal case we return here.
return n, z.err
}
// Finished file; check checksum.
if _, err := io.ReadFull(z.r, z.scratch[0:4]); err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
z.err = err
return n, z.err
}
// ZLIB (RFC 1950) is big-endian, unlike GZIP (RFC 1952).
checksum := uint32(z.scratch[0])<<24 | uint32(z.scratch[1])<<16 | uint32(z.scratch[2])<<8 | uint32(z.scratch[3])
if checksum != z.digest.Sum32() {
z.err = ErrChecksum
return n, z.err
}
return n, io.EOF
}
// Calling Close does not close the wrapped io.Reader originally passed to NewReader.
// In order for the ZLIB checksum to be verified, the reader must be
// fully consumed until the io.EOF.
func (z *reader) Close() error {
if z.err != nil && z.err != io.EOF {
return z.err
}
z.err = z.decompressor.Close()
return z.err
}
func (z *reader) Reset(r io.Reader, dict []byte) error {
*z = reader{decompressor: z.decompressor, digest: z.digest}
if fr, ok := r.(flate.Reader); ok {
z.r = fr
} else {
z.r = bufio.NewReader(r)
}
// Read the header (RFC 1950 section 2.2.).
_, z.err = io.ReadFull(z.r, z.scratch[0:2])
if z.err != nil {
if z.err == io.EOF {
z.err = io.ErrUnexpectedEOF
}
return z.err
}
h := uint(z.scratch[0])<<8 | uint(z.scratch[1])
if (z.scratch[0]&0x0f != zlibDeflate) || (h%31 != 0) {
z.err = ErrHeader
return z.err
}
haveDict := z.scratch[1]&0x20 != 0
if haveDict {
_, z.err = io.ReadFull(z.r, z.scratch[0:4])
if z.err != nil {
if z.err == io.EOF {
z.err = io.ErrUnexpectedEOF
}
return z.err
}
checksum := uint32(z.scratch[0])<<24 | uint32(z.scratch[1])<<16 | uint32(z.scratch[2])<<8 | uint32(z.scratch[3])
if checksum != adler32.Checksum(dict) {
z.err = ErrDictionary
return z.err
}
}
if z.decompressor == nil {
if haveDict {
z.decompressor = flate.NewReaderDict(z.r, dict)
} else {
z.decompressor = flate.NewReader(z.r)
}
} else {
z.decompressor.(flate.Resetter).Reset(z.r, dict)
}
if z.digest != nil {
z.digest.Reset()
} else {
z.digest = adler32.New()
}
return nil
}
+201
View File
@@ -0,0 +1,201 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package zlib
import (
"fmt"
"hash"
"hash/adler32"
"io"
"github.com/klauspost/compress/flate"
)
// These constants are copied from the flate package, so that code that imports
// "compress/zlib" does not also have to import "compress/flate".
const (
NoCompression = flate.NoCompression
BestSpeed = flate.BestSpeed
BestCompression = flate.BestCompression
DefaultCompression = flate.DefaultCompression
ConstantCompression = flate.ConstantCompression
HuffmanOnly = flate.HuffmanOnly
)
// A Writer takes data written to it and writes the compressed
// form of that data to an underlying writer (see NewWriter).
type Writer struct {
w io.Writer
level int
dict []byte
compressor *flate.Writer
digest hash.Hash32
err error
scratch [4]byte
wroteHeader bool
}
// NewWriter creates a new Writer.
// Writes to the returned Writer are compressed and written to w.
//
// It is the caller's responsibility to call Close on the WriteCloser when done.
// Writes may be buffered and not flushed until Close.
func NewWriter(w io.Writer) *Writer {
z, _ := NewWriterLevelDict(w, DefaultCompression, nil)
return z
}
// NewWriterLevel is like NewWriter but specifies the compression level instead
// of assuming DefaultCompression.
//
// The compression level can be DefaultCompression, NoCompression, HuffmanOnly
// or any integer value between BestSpeed and BestCompression inclusive.
// The error returned will be nil if the level is valid.
func NewWriterLevel(w io.Writer, level int) (*Writer, error) {
return NewWriterLevelDict(w, level, nil)
}
// NewWriterLevelDict is like NewWriterLevel but specifies a dictionary to
// compress with.
//
// The dictionary may be nil. If not, its contents should not be modified until
// the Writer is closed.
func NewWriterLevelDict(w io.Writer, level int, dict []byte) (*Writer, error) {
if level < HuffmanOnly || level > BestCompression {
return nil, fmt.Errorf("zlib: invalid compression level: %d", level)
}
return &Writer{
w: w,
level: level,
dict: dict,
}, nil
}
// Reset clears the state of the Writer z such that it is equivalent to its
// initial state from NewWriterLevel or NewWriterLevelDict, but instead writing
// to w.
func (z *Writer) Reset(w io.Writer) {
z.w = w
// z.level and z.dict left unchanged.
if z.compressor != nil {
z.compressor.Reset(w)
}
if z.digest != nil {
z.digest.Reset()
}
z.err = nil
z.scratch = [4]byte{}
z.wroteHeader = false
}
// writeHeader writes the ZLIB header.
func (z *Writer) writeHeader() (err error) {
z.wroteHeader = true
// ZLIB has a two-byte header (as documented in RFC 1950).
// The first four bits is the CINFO (compression info), which is 7 for the default deflate window size.
// The next four bits is the CM (compression method), which is 8 for deflate.
z.scratch[0] = 0x78
// The next two bits is the FLEVEL (compression level). The four values are:
// 0=fastest, 1=fast, 2=default, 3=best.
// The next bit, FDICT, is set if a dictionary is given.
// The final five FCHECK bits form a mod-31 checksum.
switch z.level {
case -2, 0, 1:
z.scratch[1] = 0 << 6
case 2, 3, 4, 5:
z.scratch[1] = 1 << 6
case 6, -1:
z.scratch[1] = 2 << 6
case 7, 8, 9:
z.scratch[1] = 3 << 6
default:
panic("unreachable")
}
if z.dict != nil {
z.scratch[1] |= 1 << 5
}
z.scratch[1] += uint8(31 - (uint16(z.scratch[0])<<8+uint16(z.scratch[1]))%31)
if _, err = z.w.Write(z.scratch[0:2]); err != nil {
return err
}
if z.dict != nil {
// The next four bytes are the Adler-32 checksum of the dictionary.
checksum := adler32.Checksum(z.dict)
z.scratch[0] = uint8(checksum >> 24)
z.scratch[1] = uint8(checksum >> 16)
z.scratch[2] = uint8(checksum >> 8)
z.scratch[3] = uint8(checksum >> 0)
if _, err = z.w.Write(z.scratch[0:4]); err != nil {
return err
}
}
if z.compressor == nil {
// Initialize deflater unless the Writer is being reused
// after a Reset call.
z.compressor, err = flate.NewWriterDict(z.w, z.level, z.dict)
if err != nil {
return err
}
z.digest = adler32.New()
}
return nil
}
// Write writes a compressed form of p to the underlying io.Writer. The
// compressed bytes are not necessarily flushed until the Writer is closed or
// explicitly flushed.
func (z *Writer) Write(p []byte) (n int, err error) {
if !z.wroteHeader {
z.err = z.writeHeader()
}
if z.err != nil {
return 0, z.err
}
if len(p) == 0 {
return 0, nil
}
n, err = z.compressor.Write(p)
if err != nil {
z.err = err
return
}
z.digest.Write(p)
return
}
// Flush flushes the Writer to its underlying io.Writer.
func (z *Writer) Flush() error {
if !z.wroteHeader {
z.err = z.writeHeader()
}
if z.err != nil {
return z.err
}
z.err = z.compressor.Flush()
return z.err
}
// Close closes the Writer, flushing any unwritten data to the underlying
// io.Writer, but does not close the underlying io.Writer.
func (z *Writer) Close() error {
if !z.wroteHeader {
z.err = z.writeHeader()
}
if z.err != nil {
return z.err
}
z.err = z.compressor.Close()
if z.err != nil {
return z.err
}
checksum := z.digest.Sum32()
// ZLIB (RFC 1950) is big-endian, unlike GZIP (RFC 1952).
z.scratch[0] = uint8(checksum >> 24)
z.scratch[1] = uint8(checksum >> 16)
z.scratch[2] = uint8(checksum >> 8)
z.scratch[3] = uint8(checksum >> 0)
_, z.err = z.w.Write(z.scratch[0:4])
return z.err
}
+22
View File
@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015 Klaus Post
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+145
View File
@@ -0,0 +1,145 @@
# cpuid
Package cpuid provides information about the CPU running the current program.
CPU features are detected on startup, and kept for fast access through the life of the application.
Currently x86 / x64 (AMD64) is supported, and no external C (cgo) code is used, which should make the library very easy to use.
You can access the CPU information by accessing the shared CPU variable of the cpuid library.
Package home: https://github.com/klauspost/cpuid
[![GoDoc][1]][2] [![Build Status][3]][4]
[1]: https://godoc.org/github.com/klauspost/cpuid?status.svg
[2]: https://godoc.org/github.com/klauspost/cpuid
[3]: https://travis-ci.org/klauspost/cpuid.svg
[4]: https://travis-ci.org/klauspost/cpuid
# features
## CPU Instructions
* **CMOV** (i686 CMOV)
* **NX** (NX (No-Execute) bit)
* **AMD3DNOW** (AMD 3DNOW)
* **AMD3DNOWEXT** (AMD 3DNowExt)
* **MMX** (standard MMX)
* **MMXEXT** (SSE integer functions or AMD MMX ext)
* **SSE** (SSE functions)
* **SSE2** (P4 SSE functions)
* **SSE3** (Prescott SSE3 functions)
* **SSSE3** (Conroe SSSE3 functions)
* **SSE4** (Penryn SSE4.1 functions)
* **SSE4A** (AMD Barcelona microarchitecture SSE4a instructions)
* **SSE42** (Nehalem SSE4.2 functions)
* **AVX** (AVX functions)
* **AVX2** (AVX2 functions)
* **FMA3** (Intel FMA 3)
* **FMA4** (Bulldozer FMA4 functions)
* **XOP** (Bulldozer XOP functions)
* **F16C** (Half-precision floating-point conversion)
* **BMI1** (Bit Manipulation Instruction Set 1)
* **BMI2** (Bit Manipulation Instruction Set 2)
* **TBM** (AMD Trailing Bit Manipulation)
* **LZCNT** (LZCNT instruction)
* **POPCNT** (POPCNT instruction)
* **AESNI** (Advanced Encryption Standard New Instructions)
* **CLMUL** (Carry-less Multiplication)
* **HTT** (Hyperthreading (enabled))
* **HLE** (Hardware Lock Elision)
* **RTM** (Restricted Transactional Memory)
* **RDRAND** (RDRAND instruction is available)
* **RDSEED** (RDSEED instruction is available)
* **ADX** (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
* **SHA** (Intel SHA Extensions)
* **AVX512F** (AVX-512 Foundation)
* **AVX512DQ** (AVX-512 Doubleword and Quadword Instructions)
* **AVX512IFMA** (AVX-512 Integer Fused Multiply-Add Instructions)
* **AVX512PF** (AVX-512 Prefetch Instructions)
* **AVX512ER** (AVX-512 Exponential and Reciprocal Instructions)
* **AVX512CD** (AVX-512 Conflict Detection Instructions)
* **AVX512BW** (AVX-512 Byte and Word Instructions)
* **AVX512VL** (AVX-512 Vector Length Extensions)
* **AVX512VBMI** (AVX-512 Vector Bit Manipulation Instructions)
* **MPX** (Intel MPX (Memory Protection Extensions))
* **ERMS** (Enhanced REP MOVSB/STOSB)
* **RDTSCP** (RDTSCP Instruction)
* **CX16** (CMPXCHG16B Instruction)
* **SGX** (Software Guard Extensions, with activation details)
## Performance
* **RDTSCP()** Returns current cycle count. Can be used for benchmarking.
* **SSE2SLOW** (SSE2 is supported, but usually not faster)
* **SSE3SLOW** (SSE3 is supported, but usually not faster)
* **ATOM** (Atom processor, some SSSE3 instructions are slower)
* **Cache line** (Probable size of a cache line).
* **L1, L2, L3 Cache size** on newer Intel/AMD CPUs.
## Cpu Vendor/VM
* **Intel**
* **AMD**
* **VIA**
* **Transmeta**
* **NSC**
* **KVM** (Kernel-based Virtual Machine)
* **MSVM** (Microsoft Hyper-V or Windows Virtual PC)
* **VMware**
* **XenHVM**
# installing
```go get github.com/klauspost/cpuid```
# example
```Go
package main
import (
"fmt"
"github.com/klauspost/cpuid"
)
func main() {
// Print basic CPU information:
fmt.Println("Name:", cpuid.CPU.BrandName)
fmt.Println("PhysicalCores:", cpuid.CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", cpuid.CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", cpuid.CPU.LogicalCores)
fmt.Println("Family", cpuid.CPU.Family, "Model:", cpuid.CPU.Model)
fmt.Println("Features:", cpuid.CPU.Features)
fmt.Println("Cacheline bytes:", cpuid.CPU.CacheLine)
fmt.Println("L1 Data Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L1 Instruction Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L2 Cache:", cpuid.CPU.Cache.L2, "bytes")
fmt.Println("L3 Cache:", cpuid.CPU.Cache.L3, "bytes")
// Test if we have a specific feature:
if cpuid.CPU.SSE() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
```
Sample output:
```
>go run main.go
Name: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz
PhysicalCores: 2
ThreadsPerCore: 2
LogicalCores: 4
Family 6 Model: 42
Features: CMOV,MMX,MMXEXT,SSE,SSE2,SSE3,SSSE3,SSE4.1,SSE4.2,AVX,AESNI,CLMUL
Cacheline bytes: 64
We have Streaming SIMD Extensions
```
# private package
In the "private" folder you can find an autogenerated version of the library you can include in your own packages.
For this purpose all exports are removed, and functions and constants are lowercased.
This is not a recommended way of using the library, but provided for convenience, if it is difficult for you to use external packages.
# license
This code is published under an MIT license. See LICENSE file for more information.
+1030
View File
File diff suppressed because it is too large. Load diff
+42
View File
@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORL CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+4(FP)
MOVL BX, ebx+8(FP)
MOVL CX, ecx+12(FP)
MOVL DX, edx+16(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func xgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+4(FP)
MOVL DX, edx+8(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET
+42
View File
@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
//+build amd64,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORQ CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmXgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+8(FP)
MOVL DX, edx+12(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET
+17
View File
@@ -0,0 +1,17 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo amd64,!gccgo
package cpuid
func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
func asmXgetbv(index uint32) (eax, edx uint32)
func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
func initCPU() {
cpuid = asmCpuid
cpuidex = asmCpuidex
xgetbv = asmXgetbv
rdtscpAsm = asmRdtscpAsm
}
+23
View File
@@ -0,0 +1,23 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build !amd64,!386 gccgo
package cpuid
func initCPU() {
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
xgetbv = func(index uint32) (eax, edx uint32) {
return 0, 0
}
rdtscpAsm = func() (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
}
+4
View File
@@ -0,0 +1,4 @@
package cpuid
//go:generate go run private-gen.go
//go:generate gofmt -w ./private
+476
View File
@@ -0,0 +1,476 @@
// +build ignore
package main
import (
"bytes"
"fmt"
"go/ast"
"go/parser"
"go/printer"
"go/token"
"io"
"io/ioutil"
"log"
"os"
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
var inFiles = []string{"cpuid.go", "cpuid_test.go"}
var copyFiles = []string{"cpuid_amd64.s", "cpuid_386.s", "detect_ref.go", "detect_intel.go"}
var fileSet = token.NewFileSet()
var reWrites = []rewrite{
initRewrite("CPUInfo -> cpuInfo"),
initRewrite("Vendor -> vendor"),
initRewrite("Flags -> flags"),
initRewrite("Detect -> detect"),
initRewrite("CPU -> cpu"),
}
var excludeNames = map[string]bool{"string": true, "join": true, "trim": true,
// cpuid_test.go
"t": true, "println": true, "logf": true, "log": true, "fatalf": true, "fatal": true,
}
var excludePrefixes = []string{"test", "benchmark"}
func main() {
Package := "private"
parserMode := parser.ParseComments
exported := make(map[string]rewrite)
for _, file := range inFiles {
in, err := os.Open(file)
if err != nil {
log.Fatalf("opening input", err)
}
src, err := ioutil.ReadAll(in)
if err != nil {
log.Fatalf("reading input", err)
}
astfile, err := parser.ParseFile(fileSet, file, src, parserMode)
if err != nil {
log.Fatalf("parsing input", err)
}
for _, rw := range reWrites {
astfile = rw(astfile)
}
// Inspect the AST and print all identifiers and literals.
var startDecl token.Pos
var endDecl token.Pos
ast.Inspect(astfile, func(n ast.Node) bool {
var s string
switch x := n.(type) {
case *ast.Ident:
if x.IsExported() {
t := strings.ToLower(x.Name)
for _, pre := range excludePrefixes {
if strings.HasPrefix(t, pre) {
return true
}
}
if excludeNames[t] != true {
//if x.Pos() > startDecl && x.Pos() < endDecl {
exported[x.Name] = initRewrite(x.Name + " -> " + t)
}
}
case *ast.GenDecl:
if x.Tok == token.CONST && x.Lparen > 0 {
startDecl = x.Lparen
endDecl = x.Rparen
// fmt.Printf("Decl:%s -> %s\n", fileSet.Position(startDecl), fileSet.Position(endDecl))
}
}
if s != "" {
fmt.Printf("%s:\t%s\n", fileSet.Position(n.Pos()), s)
}
return true
})
for _, rw := range exported {
astfile = rw(astfile)
}
var buf bytes.Buffer
printer.Fprint(&buf, fileSet, astfile)
// Remove package documentation and insert information
s := buf.String()
ind := strings.Index(buf.String(), "\npackage cpuid")
s = s[ind:]
s = "// Generated, DO NOT EDIT,\n" +
"// but copy it to your own project and rename the package.\n" +
"// See more at http://github.com/klauspost/cpuid\n" +
s
outputName := Package + string(os.PathSeparator) + file
err = ioutil.WriteFile(outputName, []byte(s), 0644)
if err != nil {
log.Fatalf("writing output: %s", err)
}
log.Println("Generated", outputName)
}
for _, file := range copyFiles {
dst := ""
if strings.HasPrefix(file, "cpuid") {
dst = Package + string(os.PathSeparator) + file
} else {
dst = Package + string(os.PathSeparator) + "cpuid_" + file
}
err := copyFile(file, dst)
if err != nil {
log.Fatalf("copying file: %s", err)
}
log.Println("Copied", dst)
}
}
// CopyFile copies a file from src to dst. If src and dst files exist, and are
// the same, then return success. Copy the file contents from src to dst.
func copyFile(src, dst string) (err error) {
sfi, err := os.Stat(src)
if err != nil {
return
}
if !sfi.Mode().IsRegular() {
// cannot copy non-regular files (e.g., directories,
// symlinks, devices, etc.)
return fmt.Errorf("CopyFile: non-regular source file %s (%q)", sfi.Name(), sfi.Mode().String())
}
dfi, err := os.Stat(dst)
if err != nil {
if !os.IsNotExist(err) {
return
}
} else {
if !(dfi.Mode().IsRegular()) {
return fmt.Errorf("CopyFile: non-regular destination file %s (%q)", dfi.Name(), dfi.Mode().String())
}
if os.SameFile(sfi, dfi) {
return
}
}
err = copyFileContents(src, dst)
return
}
// copyFileContents copies the contents of the file named src to the file named
// by dst. The file will be created if it does not already exist. If the
// destination file exists, all it's contents will be replaced by the contents
// of the source file.
func copyFileContents(src, dst string) (err error) {
in, err := os.Open(src)
if err != nil {
return
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return
}
defer func() {
cerr := out.Close()
if err == nil {
err = cerr
}
}()
if _, err = io.Copy(out, in); err != nil {
return
}
err = out.Sync()
return
}
type rewrite func(*ast.File) *ast.File
// Mostly copied from gofmt
func initRewrite(rewriteRule string) rewrite {
f := strings.Split(rewriteRule, "->")
if len(f) != 2 {
fmt.Fprintf(os.Stderr, "rewrite rule must be of the form 'pattern -> replacement'\n")
os.Exit(2)
}
pattern := parseExpr(f[0], "pattern")
replace := parseExpr(f[1], "replacement")
return func(p *ast.File) *ast.File { return rewriteFile(pattern, replace, p) }
}
// parseExpr parses s as an expression.
// It might make sense to expand this to allow statement patterns,
// but there are problems with preserving formatting and also
// with what a wildcard for a statement looks like.
func parseExpr(s, what string) ast.Expr {
x, err := parser.ParseExpr(s)
if err != nil {
fmt.Fprintf(os.Stderr, "parsing %s %s at %s\n", what, s, err)
os.Exit(2)
}
return x
}
// Keep this function for debugging.
/*
func dump(msg string, val reflect.Value) {
fmt.Printf("%s:\n", msg)
ast.Print(fileSet, val.Interface())
fmt.Println()
}
*/
// rewriteFile applies the rewrite rule 'pattern -> replace' to an entire file.
func rewriteFile(pattern, replace ast.Expr, p *ast.File) *ast.File {
cmap := ast.NewCommentMap(fileSet, p, p.Comments)
m := make(map[string]reflect.Value)
pat := reflect.ValueOf(pattern)
repl := reflect.ValueOf(replace)
var rewriteVal func(val reflect.Value) reflect.Value
rewriteVal = func(val reflect.Value) reflect.Value {
// don't bother if val is invalid to start with
if !val.IsValid() {
return reflect.Value{}
}
for k := range m {
delete(m, k)
}
val = apply(rewriteVal, val)
if match(m, pat, val) {
val = subst(m, repl, reflect.ValueOf(val.Interface().(ast.Node).Pos()))
}
return val
}
r := apply(rewriteVal, reflect.ValueOf(p)).Interface().(*ast.File)
r.Comments = cmap.Filter(r).Comments() // recreate comments list
return r
}
// set is a wrapper for x.Set(y); it protects the caller from panics if x cannot be changed to y.
func set(x, y reflect.Value) {
// don't bother if x cannot be set or y is invalid
if !x.CanSet() || !y.IsValid() {
return
}
defer func() {
if x := recover(); x != nil {
if s, ok := x.(string); ok &&
(strings.Contains(s, "type mismatch") || strings.Contains(s, "not assignable")) {
// x cannot be set to y - ignore this rewrite
return
}
panic(x)
}
}()
x.Set(y)
}
// Values/types for special cases.
var (
objectPtrNil = reflect.ValueOf((*ast.Object)(nil))
scopePtrNil = reflect.ValueOf((*ast.Scope)(nil))
identType = reflect.TypeOf((*ast.Ident)(nil))
objectPtrType = reflect.TypeOf((*ast.Object)(nil))
positionType = reflect.TypeOf(token.NoPos)
callExprType = reflect.TypeOf((*ast.CallExpr)(nil))
scopePtrType = reflect.TypeOf((*ast.Scope)(nil))
)
// apply replaces each AST field x in val with f(x), returning val.
// To avoid extra conversions, f operates on the reflect.Value form.
func apply(f func(reflect.Value) reflect.Value, val reflect.Value) reflect.Value {
if !val.IsValid() {
return reflect.Value{}
}
// *ast.Objects introduce cycles and are likely incorrect after
// rewrite; don't follow them but replace with nil instead
if val.Type() == objectPtrType {
return objectPtrNil
}
// similarly for scopes: they are likely incorrect after a rewrite;
// replace them with nil
if val.Type() == scopePtrType {
return scopePtrNil
}
switch v := reflect.Indirect(val); v.Kind() {
case reflect.Slice:
for i := 0; i < v.Len(); i++ {
e := v.Index(i)
set(e, f(e))
}
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
e := v.Field(i)
set(e, f(e))
}
case reflect.Interface:
e := v.Elem()
set(v, f(e))
}
return val
}
func isWildcard(s string) bool {
rune, size := utf8.DecodeRuneInString(s)
return size == len(s) && unicode.IsLower(rune)
}
// match returns true if pattern matches val,
// recording wildcard submatches in m.
// If m == nil, match checks whether pattern == val.
func match(m map[string]reflect.Value, pattern, val reflect.Value) bool {
// Wildcard matches any expression. If it appears multiple
// times in the pattern, it must match the same expression
// each time.
if m != nil && pattern.IsValid() && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) && val.IsValid() {
// wildcards only match valid (non-nil) expressions.
if _, ok := val.Interface().(ast.Expr); ok && !val.IsNil() {
if old, ok := m[name]; ok {
return match(nil, old, val)
}
m[name] = val
return true
}
}
}
// Otherwise, pattern and val must match recursively.
if !pattern.IsValid() || !val.IsValid() {
return !pattern.IsValid() && !val.IsValid()
}
if pattern.Type() != val.Type() {
return false
}
// Special cases.
switch pattern.Type() {
case identType:
// For identifiers, only the names need to match
// (and none of the other *ast.Object information).
// This is a common case, handle it all here instead
// of recursing down any further via reflection.
p := pattern.Interface().(*ast.Ident)
v := val.Interface().(*ast.Ident)
return p == nil && v == nil || p != nil && v != nil && p.Name == v.Name
case objectPtrType, positionType:
// object pointers and token positions always match
return true
case callExprType:
// For calls, the Ellipsis fields (token.Position) must
// match since that is how f(x) and f(x...) are different.
// Check them here but fall through for the remaining fields.
p := pattern.Interface().(*ast.CallExpr)
v := val.Interface().(*ast.CallExpr)
if p.Ellipsis.IsValid() != v.Ellipsis.IsValid() {
return false
}
}
p := reflect.Indirect(pattern)
v := reflect.Indirect(val)
if !p.IsValid() || !v.IsValid() {
return !p.IsValid() && !v.IsValid()
}
switch p.Kind() {
case reflect.Slice:
if p.Len() != v.Len() {
return false
}
for i := 0; i < p.Len(); i++ {
if !match(m, p.Index(i), v.Index(i)) {
return false
}
}
return true
case reflect.Struct:
for i := 0; i < p.NumField(); i++ {
if !match(m, p.Field(i), v.Field(i)) {
return false
}
}
return true
case reflect.Interface:
return match(m, p.Elem(), v.Elem())
}
// Handle token integers, etc.
return p.Interface() == v.Interface()
}
// subst returns a copy of pattern with values from m substituted in place
// of wildcards and pos used as the position of tokens from the pattern.
// if m == nil, subst returns a copy of pattern and doesn't change the line
// number information.
func subst(m map[string]reflect.Value, pattern reflect.Value, pos reflect.Value) reflect.Value {
if !pattern.IsValid() {
return reflect.Value{}
}
// Wildcard gets replaced with map value.
if m != nil && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) {
if old, ok := m[name]; ok {
return subst(nil, old, reflect.Value{})
}
}
}
if pos.IsValid() && pattern.Type() == positionType {
// use new position only if old position was valid in the first place
if old := pattern.Interface().(token.Pos); !old.IsValid() {
return pattern
}
return pos
}
// Otherwise copy.
switch p := pattern; p.Kind() {
case reflect.Slice:
v := reflect.MakeSlice(p.Type(), p.Len(), p.Len())
for i := 0; i < p.Len(); i++ {
v.Index(i).Set(subst(m, p.Index(i), pos))
}
return v
case reflect.Struct:
v := reflect.New(p.Type()).Elem()
for i := 0; i < p.NumField(); i++ {
v.Field(i).Set(subst(m, p.Field(i), pos))
}
return v
case reflect.Ptr:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos).Addr())
}
return v
case reflect.Interface:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos))
}
return v
}
return pattern
}
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2016 Yasuhiro Matsumoto
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+48
View File
@@ -0,0 +1,48 @@
# go-colorable
[![Godoc Reference](https://godoc.org/github.com/mattn/go-colorable?status.svg)](http://godoc.org/github.com/mattn/go-colorable)
[![Build Status](https://travis-ci.org/mattn/go-colorable.svg?branch=master)](https://travis-ci.org/mattn/go-colorable)
[![Coverage Status](https://coveralls.io/repos/github/mattn/go-colorable/badge.svg?branch=master)](https://coveralls.io/github/mattn/go-colorable?branch=master)
[![Go Report Card](https://goreportcard.com/badge/mattn/go-colorable)](https://goreportcard.com/report/mattn/go-colorable)
Colorable writer for windows.
For example, most of logger packages doesn't show colors on windows. (I know we can do it with ansicon. But I don't want.)
This package is possible to handle escape sequence for ansi color on windows.
## Too Bad!
![](https://raw.githubusercontent.com/mattn/go-colorable/gh-pages/bad.png)
## So Good!
![](https://raw.githubusercontent.com/mattn/go-colorable/gh-pages/good.png)
## Usage
```go
logrus.SetFormatter(&logrus.TextFormatter{ForceColors: true})
logrus.SetOutput(colorable.NewColorableStdout())
logrus.Info("succeeded")
logrus.Warn("not correct")
logrus.Error("something error")
logrus.Fatal("panic")
```
You can compile above code on non-windows OSs.
## Installation
```
$ go get github.com/mattn/go-colorable
```
# License
MIT
# Author
Yasuhiro Matsumoto (a.k.a mattn)
+29
View File
@@ -0,0 +1,29 @@
// +build appengine
package colorable
import (
"io"
"os"
_ "github.com/mattn/go-isatty"
)
// NewColorable return new instance of Writer which handle escape sequence.
func NewColorable(file *os.File) io.Writer {
if file == nil {
panic("nil passed instead of *os.File to NewColorable()")
}
return file
}
// NewColorableStdout return new instance of Writer which handle escape sequence for stdout.
func NewColorableStdout() io.Writer {
return os.Stdout
}
// NewColorableStderr return new instance of Writer which handle escape sequence for stderr.
func NewColorableStderr() io.Writer {
return os.Stderr
}
+30
View File
@@ -0,0 +1,30 @@
// +build !windows
// +build !appengine
package colorable
import (
"io"
"os"
_ "github.com/mattn/go-isatty"
)
// NewColorable return new instance of Writer which handle escape sequence.
func NewColorable(file *os.File) io.Writer {
if file == nil {
panic("nil passed instead of *os.File to NewColorable()")
}
return file
}
// NewColorableStdout return new instance of Writer which handle escape sequence for stdout.
func NewColorableStdout() io.Writer {
return os.Stdout
}
// NewColorableStderr return new instance of Writer which handle escape sequence for stderr.
func NewColorableStderr() io.Writer {
return os.Stderr
}
+978
View File
@@ -0,0 +1,978 @@
// +build windows
// +build !appengine
package colorable
import (
"bytes"
"io"
"math"
"os"
"strconv"
"strings"
"syscall"
"unsafe"
"github.com/mattn/go-isatty"
)
const (
foregroundBlue = 0x1
foregroundGreen = 0x2
foregroundRed = 0x4
foregroundIntensity = 0x8
foregroundMask = (foregroundRed | foregroundBlue | foregroundGreen | foregroundIntensity)
backgroundBlue = 0x10
backgroundGreen = 0x20
backgroundRed = 0x40
backgroundIntensity = 0x80
backgroundMask = (backgroundRed | backgroundBlue | backgroundGreen | backgroundIntensity)
)
const (
genericRead = 0x80000000
genericWrite = 0x40000000
)
const (
consoleTextmodeBuffer = 0x1
)
type wchar uint16
type short int16
type dword uint32
type word uint16
type coord struct {
x short
y short
}
type smallRect struct {
left short
top short
right short
bottom short
}
type consoleScreenBufferInfo struct {
size coord
cursorPosition coord
attributes word
window smallRect
maximumWindowSize coord
}
type consoleCursorInfo struct {
size dword
visible int32
}
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procGetConsoleScreenBufferInfo = kernel32.NewProc("GetConsoleScreenBufferInfo")
procSetConsoleTextAttribute = kernel32.NewProc("SetConsoleTextAttribute")
procSetConsoleCursorPosition = kernel32.NewProc("SetConsoleCursorPosition")
procFillConsoleOutputCharacter = kernel32.NewProc("FillConsoleOutputCharacterW")
procFillConsoleOutputAttribute = kernel32.NewProc("FillConsoleOutputAttribute")
procGetConsoleCursorInfo = kernel32.NewProc("GetConsoleCursorInfo")
procSetConsoleCursorInfo = kernel32.NewProc("SetConsoleCursorInfo")
procSetConsoleTitle = kernel32.NewProc("SetConsoleTitleW")
procCreateConsoleScreenBuffer = kernel32.NewProc("CreateConsoleScreenBuffer")
)
// Writer provide colorable Writer to the console
type Writer struct {
out io.Writer
handle syscall.Handle
althandle syscall.Handle
oldattr word
oldpos coord
rest bytes.Buffer
}
// NewColorable return new instance of Writer which handle escape sequence from File.
func NewColorable(file *os.File) io.Writer {
if file == nil {
panic("nil passed instead of *os.File to NewColorable()")
}
if isatty.IsTerminal(file.Fd()) {
var csbi consoleScreenBufferInfo
handle := syscall.Handle(file.Fd())
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
return &Writer{out: file, handle: handle, oldattr: csbi.attributes, oldpos: coord{0, 0}}
}
return file
}
// NewColorableStdout return new instance of Writer which handle escape sequence for stdout.
func NewColorableStdout() io.Writer {
return NewColorable(os.Stdout)
}
// NewColorableStderr return new instance of Writer which handle escape sequence for stderr.
func NewColorableStderr() io.Writer {
return NewColorable(os.Stderr)
}
var color256 = map[int]int{
0: 0x000000,
1: 0x800000,
2: 0x008000,
3: 0x808000,
4: 0x000080,
5: 0x800080,
6: 0x008080,
7: 0xc0c0c0,
8: 0x808080,
9: 0xff0000,
10: 0x00ff00,
11: 0xffff00,
12: 0x0000ff,
13: 0xff00ff,
14: 0x00ffff,
15: 0xffffff,
16: 0x000000,
17: 0x00005f,
18: 0x000087,
19: 0x0000af,
20: 0x0000d7,
21: 0x0000ff,
22: 0x005f00,
23: 0x005f5f,
24: 0x005f87,
25: 0x005faf,
26: 0x005fd7,
27: 0x005fff,
28: 0x008700,
29: 0x00875f,
30: 0x008787,
31: 0x0087af,
32: 0x0087d7,
33: 0x0087ff,
34: 0x00af00,
35: 0x00af5f,
36: 0x00af87,
37: 0x00afaf,
38: 0x00afd7,
39: 0x00afff,
40: 0x00d700,
41: 0x00d75f,
42: 0x00d787,
43: 0x00d7af,
44: 0x00d7d7,
45: 0x00d7ff,
46: 0x00ff00,
47: 0x00ff5f,
48: 0x00ff87,
49: 0x00ffaf,
50: 0x00ffd7,
51: 0x00ffff,
52: 0x5f0000,
53: 0x5f005f,
54: 0x5f0087,
55: 0x5f00af,
56: 0x5f00d7,
57: 0x5f00ff,
58: 0x5f5f00,
59: 0x5f5f5f,
60: 0x5f5f87,
61: 0x5f5faf,
62: 0x5f5fd7,
63: 0x5f5fff,
64: 0x5f8700,
65: 0x5f875f,
66: 0x5f8787,
67: 0x5f87af,
68: 0x5f87d7,
69: 0x5f87ff,
70: 0x5faf00,
71: 0x5faf5f,
72: 0x5faf87,
73: 0x5fafaf,
74: 0x5fafd7,
75: 0x5fafff,
76: 0x5fd700,
77: 0x5fd75f,
78: 0x5fd787,
79: 0x5fd7af,
80: 0x5fd7d7,
81: 0x5fd7ff,
82: 0x5fff00,
83: 0x5fff5f,
84: 0x5fff87,
85: 0x5fffaf,
86: 0x5fffd7,
87: 0x5fffff,
88: 0x870000,
89: 0x87005f,
90: 0x870087,
91: 0x8700af,
92: 0x8700d7,
93: 0x8700ff,
94: 0x875f00,
95: 0x875f5f,
96: 0x875f87,
97: 0x875faf,
98: 0x875fd7,
99: 0x875fff,
100: 0x878700,
101: 0x87875f,
102: 0x878787,
103: 0x8787af,
104: 0x8787d7,
105: 0x8787ff,
106: 0x87af00,
107: 0x87af5f,
108: 0x87af87,
109: 0x87afaf,
110: 0x87afd7,
111: 0x87afff,
112: 0x87d700,
113: 0x87d75f,
114: 0x87d787,
115: 0x87d7af,
116: 0x87d7d7,
117: 0x87d7ff,
118: 0x87ff00,
119: 0x87ff5f,
120: 0x87ff87,
121: 0x87ffaf,
122: 0x87ffd7,
123: 0x87ffff,
124: 0xaf0000,
125: 0xaf005f,
126: 0xaf0087,
127: 0xaf00af,
128: 0xaf00d7,
129: 0xaf00ff,
130: 0xaf5f00,
131: 0xaf5f5f,
132: 0xaf5f87,
133: 0xaf5faf,
134: 0xaf5fd7,
135: 0xaf5fff,
136: 0xaf8700,
137: 0xaf875f,
138: 0xaf8787,
139: 0xaf87af,
140: 0xaf87d7,
141: 0xaf87ff,
142: 0xafaf00,
143: 0xafaf5f,
144: 0xafaf87,
145: 0xafafaf,
146: 0xafafd7,
147: 0xafafff,
148: 0xafd700,
149: 0xafd75f,
150: 0xafd787,
151: 0xafd7af,
152: 0xafd7d7,
153: 0xafd7ff,
154: 0xafff00,
155: 0xafff5f,
156: 0xafff87,
157: 0xafffaf,
158: 0xafffd7,
159: 0xafffff,
160: 0xd70000,
161: 0xd7005f,
162: 0xd70087,
163: 0xd700af,
164: 0xd700d7,
165: 0xd700ff,
166: 0xd75f00,
167: 0xd75f5f,
168: 0xd75f87,
169: 0xd75faf,
170: 0xd75fd7,
171: 0xd75fff,
172: 0xd78700,
173: 0xd7875f,
174: 0xd78787,
175: 0xd787af,
176: 0xd787d7,
177: 0xd787ff,
178: 0xd7af00,
179: 0xd7af5f,
180: 0xd7af87,
181: 0xd7afaf,
182: 0xd7afd7,
183: 0xd7afff,
184: 0xd7d700,
185: 0xd7d75f,
186: 0xd7d787,
187: 0xd7d7af,
188: 0xd7d7d7,
189: 0xd7d7ff,
190: 0xd7ff00,
191: 0xd7ff5f,
192: 0xd7ff87,
193: 0xd7ffaf,
194: 0xd7ffd7,
195: 0xd7ffff,
196: 0xff0000,
197: 0xff005f,
198: 0xff0087,
199: 0xff00af,
200: 0xff00d7,
201: 0xff00ff,
202: 0xff5f00,
203: 0xff5f5f,
204: 0xff5f87,
205: 0xff5faf,
206: 0xff5fd7,
207: 0xff5fff,
208: 0xff8700,
209: 0xff875f,
210: 0xff8787,
211: 0xff87af,
212: 0xff87d7,
213: 0xff87ff,
214: 0xffaf00,
215: 0xffaf5f,
216: 0xffaf87,
217: 0xffafaf,
218: 0xffafd7,
219: 0xffafff,
220: 0xffd700,
221: 0xffd75f,
222: 0xffd787,
223: 0xffd7af,
224: 0xffd7d7,
225: 0xffd7ff,
226: 0xffff00,
227: 0xffff5f,
228: 0xffff87,
229: 0xffffaf,
230: 0xffffd7,
231: 0xffffff,
232: 0x080808,
233: 0x121212,
234: 0x1c1c1c,
235: 0x262626,
236: 0x303030,
237: 0x3a3a3a,
238: 0x444444,
239: 0x4e4e4e,
240: 0x585858,
241: 0x626262,
242: 0x6c6c6c,
243: 0x767676,
244: 0x808080,
245: 0x8a8a8a,
246: 0x949494,
247: 0x9e9e9e,
248: 0xa8a8a8,
249: 0xb2b2b2,
250: 0xbcbcbc,
251: 0xc6c6c6,
252: 0xd0d0d0,
253: 0xdadada,
254: 0xe4e4e4,
255: 0xeeeeee,
}
// `\033]0;TITLESTR\007`
func doTitleSequence(er *bytes.Reader) error {
var c byte
var err error
c, err = er.ReadByte()
if err != nil {
return err
}
if c != '0' && c != '2' {
return nil
}
c, err = er.ReadByte()
if err != nil {
return err
}
if c != ';' {
return nil
}
title := make([]byte, 0, 80)
for {
c, err = er.ReadByte()
if err != nil {
return err
}
if c == 0x07 || c == '\n' {
break
}
title = append(title, c)
}
if len(title) > 0 {
title8, err := syscall.UTF16PtrFromString(string(title))
if err == nil {
procSetConsoleTitle.Call(uintptr(unsafe.Pointer(title8)))
}
}
return nil
}
// Write write data on console
func (w *Writer) Write(data []byte) (n int, err error) {
var csbi consoleScreenBufferInfo
procGetConsoleScreenBufferInfo.Call(uintptr(w.handle), uintptr(unsafe.Pointer(&csbi)))
handle := w.handle
var er *bytes.Reader
if w.rest.Len() > 0 {
var rest bytes.Buffer
w.rest.WriteTo(&rest)
w.rest.Reset()
rest.Write(data)
er = bytes.NewReader(rest.Bytes())
} else {
er = bytes.NewReader(data)
}
var bw [1]byte
loop:
for {
c1, err := er.ReadByte()
if err != nil {
break loop
}
if c1 != 0x1b {
bw[0] = c1
w.out.Write(bw[:])
continue
}
c2, err := er.ReadByte()
if err != nil {
break loop
}
switch c2 {
case ']':
w.rest.WriteByte(c1)
w.rest.WriteByte(c2)
er.WriteTo(&w.rest)
if bytes.IndexByte(w.rest.Bytes(), 0x07) == -1 {
break loop
}
er = bytes.NewReader(w.rest.Bytes()[2:])
err := doTitleSequence(er)
if err != nil {
break loop
}
w.rest.Reset()
continue
// https://github.com/mattn/go-colorable/issues/27
case '7':
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
w.oldpos = csbi.cursorPosition
continue
case '8':
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&w.oldpos)))
continue
case 0x5b:
// execute part after switch
default:
continue
}
w.rest.WriteByte(c1)
w.rest.WriteByte(c2)
er.WriteTo(&w.rest)
var buf bytes.Buffer
var m byte
for i, c := range w.rest.Bytes()[2:] {
if ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z') || c == '@' {
m = c
er = bytes.NewReader(w.rest.Bytes()[2+i+1:])
w.rest.Reset()
break
}
buf.Write([]byte(string(c)))
}
if m == 0 {
break loop
}
switch m {
case 'A':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.y -= short(n)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'B':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.y += short(n)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'C':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.x += short(n)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'D':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.x -= short(n)
if csbi.cursorPosition.x < 0 {
csbi.cursorPosition.x = 0
}
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'E':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.x = 0
csbi.cursorPosition.y += short(n)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'F':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.x = 0
csbi.cursorPosition.y -= short(n)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'G':
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
csbi.cursorPosition.x = short(n - 1)
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'H', 'f':
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
if buf.Len() > 0 {
token := strings.Split(buf.String(), ";")
switch len(token) {
case 1:
n1, err := strconv.Atoi(token[0])
if err != nil {
continue
}
csbi.cursorPosition.y = short(n1 - 1)
case 2:
n1, err := strconv.Atoi(token[0])
if err != nil {
continue
}
n2, err := strconv.Atoi(token[1])
if err != nil {
continue
}
csbi.cursorPosition.x = short(n2 - 1)
csbi.cursorPosition.y = short(n1 - 1)
}
} else {
csbi.cursorPosition.y = 0
}
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&csbi.cursorPosition)))
case 'J':
n := 0
if buf.Len() > 0 {
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
}
var count, written dword
var cursor coord
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
switch n {
case 0:
cursor = coord{x: csbi.cursorPosition.x, y: csbi.cursorPosition.y}
count = dword(csbi.size.x - csbi.cursorPosition.x + (csbi.size.y-csbi.cursorPosition.y)*csbi.size.x)
case 1:
cursor = coord{x: csbi.window.left, y: csbi.window.top}
count = dword(csbi.size.x - csbi.cursorPosition.x + (csbi.window.top-csbi.cursorPosition.y)*csbi.size.x)
case 2:
cursor = coord{x: csbi.window.left, y: csbi.window.top}
count = dword(csbi.size.x - csbi.cursorPosition.x + (csbi.size.y-csbi.cursorPosition.y)*csbi.size.x)
}
procFillConsoleOutputCharacter.Call(uintptr(handle), uintptr(' '), uintptr(count), *(*uintptr)(unsafe.Pointer(&cursor)), uintptr(unsafe.Pointer(&written)))
procFillConsoleOutputAttribute.Call(uintptr(handle), uintptr(csbi.attributes), uintptr(count), *(*uintptr)(unsafe.Pointer(&cursor)), uintptr(unsafe.Pointer(&written)))
case 'K':
n := 0
if buf.Len() > 0 {
n, err = strconv.Atoi(buf.String())
if err != nil {
continue
}
}
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
var cursor coord
var count, written dword
switch n {
case 0:
cursor = coord{x: csbi.cursorPosition.x, y: csbi.cursorPosition.y}
count = dword(csbi.size.x - csbi.cursorPosition.x)
case 1:
cursor = coord{x: csbi.window.left, y: csbi.window.top + csbi.cursorPosition.y}
count = dword(csbi.size.x - csbi.cursorPosition.x)
case 2:
cursor = coord{x: csbi.window.left, y: csbi.window.top + csbi.cursorPosition.y}
count = dword(csbi.size.x)
}
procFillConsoleOutputCharacter.Call(uintptr(handle), uintptr(' '), uintptr(count), *(*uintptr)(unsafe.Pointer(&cursor)), uintptr(unsafe.Pointer(&written)))
procFillConsoleOutputAttribute.Call(uintptr(handle), uintptr(csbi.attributes), uintptr(count), *(*uintptr)(unsafe.Pointer(&cursor)), uintptr(unsafe.Pointer(&written)))
case 'm':
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
attr := csbi.attributes
cs := buf.String()
if cs == "" {
procSetConsoleTextAttribute.Call(uintptr(handle), uintptr(w.oldattr))
continue
}
token := strings.Split(cs, ";")
for i := 0; i < len(token); i++ {
ns := token[i]
if n, err = strconv.Atoi(ns); err == nil {
switch {
case n == 0 || n == 100:
attr = w.oldattr
case 1 <= n && n <= 5:
attr |= foregroundIntensity
case n == 7:
attr = ((attr & foregroundMask) << 4) | ((attr & backgroundMask) >> 4)
case n == 22 || n == 25:
attr |= foregroundIntensity
case n == 27:
attr = ((attr & foregroundMask) << 4) | ((attr & backgroundMask) >> 4)
case 30 <= n && n <= 37:
attr &= backgroundMask
if (n-30)&1 != 0 {
attr |= foregroundRed
}
if (n-30)&2 != 0 {
attr |= foregroundGreen
}
if (n-30)&4 != 0 {
attr |= foregroundBlue
}
case n == 38: // set foreground color.
if i < len(token)-2 && (token[i+1] == "5" || token[i+1] == "05") {
if n256, err := strconv.Atoi(token[i+2]); err == nil {
if n256foreAttr == nil {
n256setup()
}
attr &= backgroundMask
attr |= n256foreAttr[n256]
i += 2
}
} else if len(token) == 5 && token[i+1] == "2" {
var r, g, b int
r, _ = strconv.Atoi(token[i+2])
g, _ = strconv.Atoi(token[i+3])
b, _ = strconv.Atoi(token[i+4])
i += 4
if r > 127 {
attr |= foregroundRed
}
if g > 127 {
attr |= foregroundGreen
}
if b > 127 {
attr |= foregroundBlue
}
} else {
attr = attr & (w.oldattr & backgroundMask)
}
case n == 39: // reset foreground color.
attr &= backgroundMask
attr |= w.oldattr & foregroundMask
case 40 <= n && n <= 47:
attr &= foregroundMask
if (n-40)&1 != 0 {
attr |= backgroundRed
}
if (n-40)&2 != 0 {
attr |= backgroundGreen
}
if (n-40)&4 != 0 {
attr |= backgroundBlue
}
case n == 48: // set background color.
if i < len(token)-2 && token[i+1] == "5" {
if n256, err := strconv.Atoi(token[i+2]); err == nil {
if n256backAttr == nil {
n256setup()
}
attr &= foregroundMask
attr |= n256backAttr[n256]
i += 2
}
} else if len(token) == 5 && token[i+1] == "2" {
var r, g, b int
r, _ = strconv.Atoi(token[i+2])
g, _ = strconv.Atoi(token[i+3])
b, _ = strconv.Atoi(token[i+4])
i += 4
if r > 127 {
attr |= backgroundRed
}
if g > 127 {
attr |= backgroundGreen
}
if b > 127 {
attr |= backgroundBlue
}
} else {
attr = attr & (w.oldattr & foregroundMask)
}
case n == 49: // reset foreground color.
attr &= foregroundMask
attr |= w.oldattr & backgroundMask
case 90 <= n && n <= 97:
attr = (attr & backgroundMask)
attr |= foregroundIntensity
if (n-90)&1 != 0 {
attr |= foregroundRed
}
if (n-90)&2 != 0 {
attr |= foregroundGreen
}
if (n-90)&4 != 0 {
attr |= foregroundBlue
}
case 100 <= n && n <= 107:
attr = (attr & foregroundMask)
attr |= backgroundIntensity
if (n-100)&1 != 0 {
attr |= backgroundRed
}
if (n-100)&2 != 0 {
attr |= backgroundGreen
}
if (n-100)&4 != 0 {
attr |= backgroundBlue
}
}
procSetConsoleTextAttribute.Call(uintptr(handle), uintptr(attr))
}
}
case 'h':
var ci consoleCursorInfo
cs := buf.String()
if cs == "5>" {
procGetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
ci.visible = 0
procSetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
} else if cs == "?25" {
procGetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
ci.visible = 1
procSetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
} else if cs == "?1049" {
if w.althandle == 0 {
h, _, _ := procCreateConsoleScreenBuffer.Call(uintptr(genericRead|genericWrite), 0, 0, uintptr(consoleTextmodeBuffer), 0, 0)
w.althandle = syscall.Handle(h)
if w.althandle != 0 {
handle = w.althandle
}
}
}
case 'l':
var ci consoleCursorInfo
cs := buf.String()
if cs == "5>" {
procGetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
ci.visible = 1
procSetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
} else if cs == "?25" {
procGetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
ci.visible = 0
procSetConsoleCursorInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&ci)))
} else if cs == "?1049" {
if w.althandle != 0 {
syscall.CloseHandle(w.althandle)
w.althandle = 0
handle = w.handle
}
}
case 's':
procGetConsoleScreenBufferInfo.Call(uintptr(handle), uintptr(unsafe.Pointer(&csbi)))
w.oldpos = csbi.cursorPosition
case 'u':
procSetConsoleCursorPosition.Call(uintptr(handle), *(*uintptr)(unsafe.Pointer(&w.oldpos)))
}
}
return len(data), nil
}
type consoleColor struct {
rgb int
red bool
green bool
blue bool
intensity bool
}
func (c consoleColor) foregroundAttr() (attr word) {
if c.red {
attr |= foregroundRed
}
if c.green {
attr |= foregroundGreen
}
if c.blue {
attr |= foregroundBlue
}
if c.intensity {
attr |= foregroundIntensity
}
return
}
func (c consoleColor) backgroundAttr() (attr word) {
if c.red {
attr |= backgroundRed
}
if c.green {
attr |= backgroundGreen
}
if c.blue {
attr |= backgroundBlue
}
if c.intensity {
attr |= backgroundIntensity
}
return
}
var color16 = []consoleColor{
{0x000000, false, false, false, false},
{0x000080, false, false, true, false},
{0x008000, false, true, false, false},
{0x008080, false, true, true, false},
{0x800000, true, false, false, false},
{0x800080, true, false, true, false},
{0x808000, true, true, false, false},
{0xc0c0c0, true, true, true, false},
{0x808080, false, false, false, true},
{0x0000ff, false, false, true, true},
{0x00ff00, false, true, false, true},
{0x00ffff, false, true, true, true},
{0xff0000, true, false, false, true},
{0xff00ff, true, false, true, true},
{0xffff00, true, true, false, true},
{0xffffff, true, true, true, true},
}
type hsv struct {
h, s, v float32
}
func (a hsv) dist(b hsv) float32 {
dh := a.h - b.h
switch {
case dh > 0.5:
dh = 1 - dh
case dh < -0.5:
dh = -1 - dh
}
ds := a.s - b.s
dv := a.v - b.v
return float32(math.Sqrt(float64(dh*dh + ds*ds + dv*dv)))
}
func toHSV(rgb int) hsv {
r, g, b := float32((rgb&0xFF0000)>>16)/256.0,
float32((rgb&0x00FF00)>>8)/256.0,
float32(rgb&0x0000FF)/256.0
min, max := minmax3f(r, g, b)
h := max - min
if h > 0 {
if max == r {
h = (g - b) / h
if h < 0 {
h += 6
}
} else if max == g {
h = 2 + (b-r)/h
} else {
h = 4 + (r-g)/h
}
}
h /= 6.0
s := max - min
if max != 0 {
s /= max
}
v := max
return hsv{h: h, s: s, v: v}
}
type hsvTable []hsv
func toHSVTable(rgbTable []consoleColor) hsvTable {
t := make(hsvTable, len(rgbTable))
for i, c := range rgbTable {
t[i] = toHSV(c.rgb)
}
return t
}
func (t hsvTable) find(rgb int) consoleColor {
hsv := toHSV(rgb)
n := 7
l := float32(5.0)
for i, p := range t {
d := hsv.dist(p)
if d < l {
l, n = d, i
}
}
return color16[n]
}
func minmax3f(a, b, c float32) (min, max float32) {
if a < b {
if b < c {
return a, c
} else if a < c {
return a, b
} else {
return c, b
}
} else {
if a < c {
return b, c
} else if b < c {
return b, a
} else {
return c, a
}
}
}
var n256foreAttr []word
var n256backAttr []word
func n256setup() {
n256foreAttr = make([]word, 256)
n256backAttr = make([]word, 256)
t := toHSVTable(color16)
for i, rgb := range color256 {
c := t.find(rgb)
n256foreAttr[i] = c.foregroundAttr()
n256backAttr[i] = c.backgroundAttr()
}
}
+55
View File
@@ -0,0 +1,55 @@
package colorable
import (
"bytes"
"io"
)
// NonColorable hold writer but remove escape sequence.
type NonColorable struct {
out io.Writer
}
// NewNonColorable return new instance of Writer which remove escape sequence from Writer.
func NewNonColorable(w io.Writer) io.Writer {
return &NonColorable{out: w}
}
// Write write data on console
func (w *NonColorable) Write(data []byte) (n int, err error) {
er := bytes.NewReader(data)
var bw [1]byte
loop:
for {
c1, err := er.ReadByte()
if err != nil {
break loop
}
if c1 != 0x1b {
bw[0] = c1
w.out.Write(bw[:])
continue
}
c2, err := er.ReadByte()
if err != nil {
break loop
}
if c2 != 0x5b {
continue
}
var buf bytes.Buffer
for {
c, err := er.ReadByte()
if err != nil {
break loop
}
if ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z') || c == '@' {
break
}
buf.Write([]byte(string(c)))
}
}
return len(data), nil
}
+9
View File
@@ -0,0 +1,9 @@
Copyright (c) Yasuhiro MATSUMOTO <mattn.jp@gmail.com>
MIT License (Expat)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+50
View File
@@ -0,0 +1,50 @@
# go-isatty
[![Godoc Reference](https://godoc.org/github.com/mattn/go-isatty?status.svg)](http://godoc.org/github.com/mattn/go-isatty)
[![Build Status](https://travis-ci.org/mattn/go-isatty.svg?branch=master)](https://travis-ci.org/mattn/go-isatty)
[![Coverage Status](https://coveralls.io/repos/github/mattn/go-isatty/badge.svg?branch=master)](https://coveralls.io/github/mattn/go-isatty?branch=master)
[![Go Report Card](https://goreportcard.com/badge/mattn/go-isatty)](https://goreportcard.com/report/mattn/go-isatty)
isatty for golang
## Usage
```go
package main
import (
"fmt"
"github.com/mattn/go-isatty"
"os"
)
func main() {
if isatty.IsTerminal(os.Stdout.Fd()) {
fmt.Println("Is Terminal")
} else if isatty.IsCygwinTerminal(os.Stdout.Fd()) {
fmt.Println("Is Cygwin/MSYS2 Terminal")
} else {
fmt.Println("Is Not Terminal")
}
}
```
## Installation
```
$ go get github.com/mattn/go-isatty
```
## License
MIT
## Author
Yasuhiro Matsumoto (a.k.a mattn)
## Thanks
* k-takata: base idea for IsCygwinTerminal
https://github.com/k-takata/go-iscygpty
+2
View File
@@ -0,0 +1,2 @@
// Package isatty implements interface to isatty
package isatty
+15
View File
@@ -0,0 +1,15 @@
// +build appengine
package isatty
// IsTerminal returns true if the file descriptor is terminal which
// is always false on on appengine classic which is a sandboxed PaaS.
func IsTerminal(fd uintptr) bool {
return false
}
// IsCygwinTerminal() return true if the file descriptor is a cygwin or msys2
// terminal. This is also always false on this environment.
func IsCygwinTerminal(fd uintptr) bool {
return false
}
+18
View File
@@ -0,0 +1,18 @@
// +build darwin freebsd openbsd netbsd dragonfly
// +build !appengine
package isatty
import (
"syscall"
"unsafe"
)
const ioctlReadTermios = syscall.TIOCGETA
// IsTerminal return true if the file descriptor is terminal.
func IsTerminal(fd uintptr) bool {
var termios syscall.Termios
_, _, err := syscall.Syscall6(syscall.SYS_IOCTL, fd, ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0)
return err == 0
}
+18
View File
@@ -0,0 +1,18 @@
// +build linux
// +build !appengine,!ppc64,!ppc64le
package isatty
import (
"syscall"
"unsafe"
)
const ioctlReadTermios = syscall.TCGETS
// IsTerminal return true if the file descriptor is terminal.
func IsTerminal(fd uintptr) bool {
var termios syscall.Termios
_, _, err := syscall.Syscall6(syscall.SYS_IOCTL, fd, ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0)
return err == 0
}
+19
View File
@@ -0,0 +1,19 @@
// +build linux
// +build ppc64 ppc64le
package isatty
import (
"unsafe"
syscall "golang.org/x/sys/unix"
)
const ioctlReadTermios = syscall.TCGETS
// IsTerminal return true if the file descriptor is terminal.
func IsTerminal(fd uintptr) bool {
var termios syscall.Termios
_, _, err := syscall.Syscall6(syscall.SYS_IOCTL, fd, ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0)
return err == 0
}
+10
View File
@@ -0,0 +1,10 @@
// +build !windows
// +build !appengine
package isatty
// IsCygwinTerminal return true if the file descriptor is a cygwin or msys2
// terminal. This is also always false on this environment.
func IsCygwinTerminal(fd uintptr) bool {
return false
}
+16
View File
@@ -0,0 +1,16 @@
// +build solaris
// +build !appengine
package isatty
import (
"golang.org/x/sys/unix"
)
// IsTerminal returns true if the given file descriptor is a terminal.
// see: http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libbc/libc/gen/common/isatty.c
func IsTerminal(fd uintptr) bool {
var termio unix.Termio
err := unix.IoctlSetTermio(int(fd), unix.TCGETA, &termio)
return err == nil
}
+94
View File
@@ -0,0 +1,94 @@
// +build windows
// +build !appengine
package isatty
import (
"strings"
"syscall"
"unicode/utf16"
"unsafe"
)
const (
fileNameInfo uintptr = 2
fileTypePipe = 3
)
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procGetConsoleMode = kernel32.NewProc("GetConsoleMode")
procGetFileInformationByHandleEx = kernel32.NewProc("GetFileInformationByHandleEx")
procGetFileType = kernel32.NewProc("GetFileType")
)
func init() {
// Check if GetFileInformationByHandleEx is available.
if procGetFileInformationByHandleEx.Find() != nil {
procGetFileInformationByHandleEx = nil
}
}
// IsTerminal return true if the file descriptor is terminal.
func IsTerminal(fd uintptr) bool {
var st uint32
r, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, fd, uintptr(unsafe.Pointer(&st)), 0)
return r != 0 && e == 0
}
// Check pipe name is used for cygwin/msys2 pty.
// Cygwin/MSYS2 PTY has a name like:
// \{cygwin,msys}-XXXXXXXXXXXXXXXX-ptyN-{from,to}-master
func isCygwinPipeName(name string) bool {
token := strings.Split(name, "-")
if len(token) < 5 {
return false
}
if token[0] != `\msys` && token[0] != `\cygwin` {
return false
}
if token[1] == "" {
return false
}
if !strings.HasPrefix(token[2], "pty") {
return false
}
if token[3] != `from` && token[3] != `to` {
return false
}
if token[4] != "master" {
return false
}
return true
}
// IsCygwinTerminal() return true if the file descriptor is a cygwin or msys2
// terminal.
func IsCygwinTerminal(fd uintptr) bool {
if procGetFileInformationByHandleEx == nil {
return false
}
// Cygwin/msys's pty is a pipe.
ft, _, e := syscall.Syscall(procGetFileType.Addr(), 1, fd, 0, 0)
if ft != fileTypePipe || e != 0 {
return false
}
var buf [2 + syscall.MAX_PATH]uint16
r, _, e := syscall.Syscall6(procGetFileInformationByHandleEx.Addr(),
4, fd, fileNameInfo, uintptr(unsafe.Pointer(&buf)),
uintptr(len(buf)*2), 0, 0)
if r == 0 || e != 0 {
return false
}
l := *(*uint32)(unsafe.Pointer(&buf))
return isCygwinPipeName(string(utf16.Decode(buf[2 : 2+l/2])))
}
+3
View File
@@ -0,0 +1,3 @@
.idea/
*.iml
+11
View File
@@ -0,0 +1,11 @@
# Revel command line tools
Provides the `revel` command, used to create and run Revel apps.
- More info at http://revel.github.io/manual/tool.html
Install
------------
```bash
go get github.com/revel/cmd/revel
```
+128
View File
@@ -0,0 +1,128 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package harness
import (
"bytes"
"errors"
"fmt"
"io"
"os"
"os/exec"
"time"
"github.com/revel/revel"
)
// App contains the configuration for running a Revel app. (Not for the app itself)
// Its only purpose is constructing the command to execute.
type App struct {
BinaryPath string // Path to the app executable
Port int // Port to pass as a command line argument.
cmd AppCmd // The last cmd returned.
}
// NewApp returns app instance with binary path in it
func NewApp(binPath string) *App {
return &App{BinaryPath: binPath}
}
// Cmd returns a command to run the app server using the current configuration.
func (a *App) Cmd() AppCmd {
a.cmd = NewAppCmd(a.BinaryPath, a.Port)
return a.cmd
}
// Kill the last app command returned.
func (a *App) Kill() {
a.cmd.Kill()
}
// AppCmd manages the running of a Revel app server.
// It requires revel.Init to have been called previously.
type AppCmd struct {
*exec.Cmd
}
// NewAppCmd returns the AppCmd with parameters initialized for running app
func NewAppCmd(binPath string, port int) AppCmd {
cmd := exec.Command(binPath,
fmt.Sprintf("-port=%d", port),
fmt.Sprintf("-importPath=%s", revel.ImportPath),
fmt.Sprintf("-runMode=%s", revel.RunMode))
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
return AppCmd{cmd}
}
// Start the app server, and wait until it is ready to serve requests.
func (cmd AppCmd) Start() error {
listeningWriter := &startupListeningWriter{os.Stdout, make(chan bool)}
cmd.Stdout = listeningWriter
revel.RevelLog.Debug("Exec app:", "path", cmd.Path, "args", cmd.Args)
if err := cmd.Cmd.Start(); err != nil {
revel.RevelLog.Fatal("Error running:", "error", err)
}
select {
case <-cmd.waitChan():
return errors.New("revel/harness: app died")
case <-time.After(30 * time.Second):
revel.RevelLog.Debug("Killing revel server process did not respond after wait timeout", "processid", cmd.Process.Pid)
cmd.Kill()
return errors.New("revel/harness: app timed out")
case <-listeningWriter.notifyReady:
return nil
}
// TODO remove this unreachable code and document it
panic("Impossible")
}
// Run the app server inline. Never returns.
func (cmd AppCmd) Run() {
revel.RevelLog.Debug("Exec app:", "path", cmd.Path, "args", cmd.Args)
if err := cmd.Cmd.Run(); err != nil {
revel.RevelLog.Fatal("Error running:", "error", err)
}
}
// Kill terminates the app server if it's running.
func (cmd AppCmd) Kill() {
if cmd.Cmd != nil && (cmd.ProcessState == nil || !cmd.ProcessState.Exited()) {
revel.RevelLog.Debug("Killing revel server pid", "pid", cmd.Process.Pid)
err := cmd.Process.Kill()
if err != nil {
revel.RevelLog.Fatal("Failed to kill revel server:", "error", err)
}
}
}
// Return a channel that is notified when Wait() returns.
func (cmd AppCmd) waitChan() <-chan struct{} {
ch := make(chan struct{}, 1)
go func() {
_ = cmd.Wait()
ch <- struct{}{}
}()
return ch
}
// A io.Writer that copies to the destination, and listens for "Listening on.."
// in the stream. (Which tells us when the revel server has finished starting up)
// This is super ghetto, but by far the simplest thing that should work.
type startupListeningWriter struct {
dest io.Writer
notifyReady chan bool
}
func (w *startupListeningWriter) Write(p []byte) (n int, err error) {
if w.notifyReady != nil && bytes.Contains(p, []byte("Listening")) {
w.notifyReady <- true
w.notifyReady = nil
}
return w.dest.Write(p)
}
+491
View File
@@ -0,0 +1,491 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package harness
import (
"fmt"
"go/build"
"os"
"os/exec"
"path"
"path/filepath"
"regexp"
"runtime"
"strconv"
"strings"
"text/template"
"time"
"github.com/revel/revel"
)
var importErrorPattern = regexp.MustCompile("cannot find package \"([^\"]+)\"")
// Build the app:
// 1. Generate the the main.go file.
// 2. Run the appropriate "go build" command.
// Requires that revel.Init has been called previously.
// Returns the path to the built binary, and an error if there was a problem building it.
func Build(buildFlags ...string) (app *App, compileError *revel.Error) {
// First, clear the generated files (to avoid them messing with ProcessSource).
cleanSource("tmp", "routes")
sourceInfo, compileError := ProcessSource(revel.CodePaths)
if compileError != nil {
return nil, compileError
}
// Add the db.import to the import paths.
if dbImportPath, found := revel.Config.String("db.import"); found {
sourceInfo.InitImportPaths = append(sourceInfo.InitImportPaths, strings.Split(dbImportPath,",")...)
}
// Generate two source files.
templateArgs := map[string]interface{}{
"Controllers": sourceInfo.ControllerSpecs(),
"ValidationKeys": sourceInfo.ValidationKeys,
"ImportPaths": calcImportAliases(sourceInfo),
"TestSuites": sourceInfo.TestSuites(),
}
genSource("tmp", "main.go", RevelMainTemplate, templateArgs)
genSource("routes", "routes.go", RevelRoutesTemplate, templateArgs)
// Read build config.
buildTags := revel.Config.StringDefault("build.tags", "")
// Build the user program (all code under app).
// It relies on the user having "go" installed.
goPath, err := exec.LookPath("go")
if err != nil {
revel.RevelLog.Fatalf("Go executable not found in PATH.")
}
// Detect if deps tool should be used (is there a vendor folder ?)
useVendor := revel.DirExists(filepath.Join(revel.BasePath, "vendor"))
basePath := revel.BasePath
for !useVendor {
basePath = filepath.Dir(basePath)
found := false
// Check to see if we are still in the GOPATH
for _, path := range filepath.SplitList(build.Default.GOPATH) {
if strings.HasPrefix(basePath, path) {
found = true
break
}
}
if !found {
break
} else {
useVendor = revel.DirExists(filepath.Join(basePath, "vendor"))
}
}
var depPath string
if useVendor {
revel.RevelLog.Info("Vendor folder detected, scanning for deps in path")
depPath, err = exec.LookPath("dep")
if err != nil {
// Do not halt build unless a new package needs to be imported
revel.RevelLog.Warn("Build: `dep` executable not found in PATH, but vendor folder detected." +
"Packages can only be added automatically to the vendor folder using the `dep` tool. " +
"You can install the `dep` tool by doing a `go get -u github.com/golang/dep/cmd/dep`")
}
} else {
revel.RevelLog.Info("No vendor folder detected, not using dependency manager to import files")
}
pkg, err := build.Default.Import(revel.ImportPath, "", build.FindOnly)
if err != nil {
revel.RevelLog.Fatal("Failure importing", "path", revel.ImportPath)
}
// Binary path is a combination of $GOBIN/revel.d directory, app's import path and its name.
binName := filepath.Join(pkg.BinDir, "revel.d", revel.ImportPath, filepath.Base(revel.BasePath))
// Change binary path for Windows build
goos := runtime.GOOS
if goosEnv := os.Getenv("GOOS"); goosEnv != "" {
goos = goosEnv
}
if goos == "windows" {
binName += ".exe"
}
gotten := make(map[string]struct{})
for {
appVersion := getAppVersion()
buildTime := time.Now().UTC().Format(time.RFC3339)
versionLinkerFlags := fmt.Sprintf("-X %s/app.AppVersion=%s -X %s/app.BuildTime=%s",
revel.ImportPath, appVersion, revel.ImportPath, buildTime)
flags := []string{
"build",
"-i",
"-ldflags", versionLinkerFlags,
"-tags", buildTags,
"-o", binName}
// Add in build flags
flags = append(flags, buildFlags...)
// This is Go main path
// Note: It's not applicable for filepath.* usage
flags = append(flags, path.Join(revel.ImportPath, "app", "tmp"))
buildCmd := exec.Command(goPath, flags...)
revel.RevelLog.Debug("Exec:", "args", buildCmd.Args)
output, err := buildCmd.CombinedOutput()
// If the build succeeded, we're done.
if err == nil {
return NewApp(binName), nil
}
revel.RevelLog.Error(string(output))
// See if it was an import error that we can go get.
matches := importErrorPattern.FindAllStringSubmatch(string(output), -1)
if matches == nil {
return nil, newCompileError(output)
}
for _, match := range matches {
// Ensure we haven't already tried to go get it.
pkgName := match[1]
if _, alreadyTried := gotten[pkgName]; alreadyTried {
return nil, newCompileError(output)
}
gotten[pkgName] = struct{}{}
// Execute "go get <pkg>"
// Or dep `dep ensure -add <pkg>` if it is there
var getCmd *exec.Cmd
if useVendor {
if depPath == "" {
revel.RevelLog.Error("Build: Vendor folder found, but the `dep` tool was not found, " +
"if you use a different vendoring (package management) tool please add the following packages by hand, " +
"or install the `dep` tool into your gopath by doing a `go get -u github.com/golang/dep/cmd/dep`. " +
"For more information and usage of the tool please see http://github.com/golang/dep")
for _, pkg := range matches {
revel.RevelLog.Error("Missing package", "package", pkg[1])
}
}
getCmd = exec.Command(depPath, "ensure", "-add", pkgName)
} else {
getCmd = exec.Command(goPath, "get", pkgName)
}
revel.RevelLog.Debug("Exec:", "args", getCmd.Args)
getOutput, err := getCmd.CombinedOutput()
if err != nil {
revel.RevelLog.Error(string(getOutput))
return nil, newCompileError(output)
}
}
// Success getting the import, attempt to build again.
}
// TODO remove this unreachable code and document it
revel.RevelLog.Fatalf("Not reachable")
return nil, nil
}
// Try to define a version string for the compiled app
// The following is tried (first match returns):
// - Read a version explicitly specified in the APP_VERSION environment
// variable
// - Read the output of "git describe" if the source is in a git repository
// If no version can be determined, an empty string is returned.
func getAppVersion() string {
if version := os.Getenv("APP_VERSION"); version != "" {
return version
}
// Check for the git binary
if gitPath, err := exec.LookPath("git"); err == nil {
// Check for the .git directory
gitDir := filepath.Join(revel.BasePath, ".git")
info, err := os.Stat(gitDir)
if (err != nil && os.IsNotExist(err)) || !info.IsDir() {
return ""
}
gitCmd := exec.Command(gitPath, "--git-dir="+gitDir, "describe", "--always", "--dirty")
revel.RevelLog.Debug("Exec:", "args", gitCmd.Args)
output, err := gitCmd.Output()
if err != nil {
revel.RevelLog.Warn("Cannot determine git repository version:", "error", err)
return ""
}
return "git-" + strings.TrimSpace(string(output))
}
return ""
}
func cleanSource(dirs ...string) {
for _, dir := range dirs {
cleanDir(dir)
}
}
func cleanDir(dir string) {
revel.RevelLog.Info("Cleaning dir " + dir)
tmpPath := filepath.Join(revel.AppPath, dir)
f, err := os.Open(tmpPath)
if err != nil {
if !os.IsNotExist(err) {
revel.RevelLog.Error("Failed to clean dir:", "error", err)
}
} else {
defer func() {
_ = f.Close()
}()
infos, err := f.Readdir(0)
if err != nil {
if !os.IsNotExist(err) {
revel.RevelLog.Error("Failed to clean dir:", "error", err)
}
} else {
for _, info := range infos {
pathName := filepath.Join(tmpPath, info.Name())
if info.IsDir() {
err := os.RemoveAll(pathName)
if err != nil {
revel.RevelLog.Error("Failed to remove dir:", "error", err)
}
} else {
err := os.Remove(pathName)
if err != nil {
revel.RevelLog.Error("Failed to remove file:", "error", err)
}
}
}
}
}
}
// genSource renders the given template to produce source code, which it writes
// to the given directory and file.
func genSource(dir, filename, templateSource string, args map[string]interface{}) {
sourceCode := revel.ExecuteTemplate(
template.Must(template.New("").Parse(templateSource)),
args)
// Create a fresh dir.
cleanSource(dir)
tmpPath := filepath.Join(revel.AppPath, dir)
err := os.Mkdir(tmpPath, 0777)
if err != nil && !os.IsExist(err) {
revel.RevelLog.Fatalf("Failed to make '%v' directory: %v", dir, err)
}
// Create the file
file, err := os.Create(filepath.Join(tmpPath, filename))
if err != nil {
revel.RevelLog.Fatalf("Failed to create file: %v", err)
}
defer func() {
_ = file.Close()
}()
if _, err = file.WriteString(sourceCode); err != nil {
revel.RevelLog.Fatalf("Failed to write to file: %v", err)
}
}
// Looks through all the method args and returns a set of unique import paths
// that cover all the method arg types.
// Additionally, assign package aliases when necessary to resolve ambiguity.
func calcImportAliases(src *SourceInfo) map[string]string {
aliases := make(map[string]string)
typeArrays := [][]*TypeInfo{src.ControllerSpecs(), src.TestSuites()}
for _, specs := range typeArrays {
for _, spec := range specs {
addAlias(aliases, spec.ImportPath, spec.PackageName)
for _, methSpec := range spec.MethodSpecs {
for _, methArg := range methSpec.Args {
if methArg.ImportPath == "" {
continue
}
addAlias(aliases, methArg.ImportPath, methArg.TypeExpr.PkgName)
}
}
}
}
// Add the "InitImportPaths", with alias "_"
for _, importPath := range src.InitImportPaths {
if _, ok := aliases[importPath]; !ok {
aliases[importPath] = "_"
}
}
return aliases
}
func addAlias(aliases map[string]string, importPath, pkgName string) {
alias, ok := aliases[importPath]
if ok {
return
}
alias = makePackageAlias(aliases, pkgName)
aliases[importPath] = alias
}
func makePackageAlias(aliases map[string]string, pkgName string) string {
i := 0
alias := pkgName
for containsValue(aliases, alias) || alias == "revel" {
alias = fmt.Sprintf("%s%d", pkgName, i)
i++
}
return alias
}
func containsValue(m map[string]string, val string) bool {
for _, v := range m {
if v == val {
return true
}
}
return false
}
// Parse the output of the "go build" command.
// Return a detailed Error.
func newCompileError(output []byte) *revel.Error {
errorMatch := regexp.MustCompile(`(?m)^([^:#]+):(\d+):(\d+:)? (.*)$`).
FindSubmatch(output)
if errorMatch == nil {
errorMatch = regexp.MustCompile(`(?m)^(.*?)\:(\d+)\:\s(.*?)$`).FindSubmatch(output)
if errorMatch == nil {
revel.RevelLog.Error("Failed to parse build errors", "error", string(output))
return &revel.Error{
SourceType: "Go code",
Title: "Go Compilation Error",
Description: "See console for build error.",
}
}
errorMatch = append(errorMatch, errorMatch[3])
revel.RevelLog.Error("Build errors", "errors", string(output))
}
// Read the source for the offending file.
var (
relFilename = string(errorMatch[1]) // e.g. "src/revel/sample/app/controllers/app.go"
absFilename, _ = filepath.Abs(relFilename)
line, _ = strconv.Atoi(string(errorMatch[2]))
description = string(errorMatch[4])
compileError = &revel.Error{
SourceType: "Go code",
Title: "Go Compilation Error",
Path: relFilename,
Description: description,
Line: line,
}
)
errorLink := revel.Config.StringDefault("error.link", "")
if errorLink != "" {
compileError.SetLink(errorLink)
}
fileStr, err := revel.ReadLines(absFilename)
if err != nil {
compileError.MetaError = absFilename + ": " + err.Error()
revel.RevelLog.Error(compileError.MetaError)
return compileError
}
compileError.SourceLines = fileStr
return compileError
}
// RevelMainTemplate template for app/tmp/main.go
const RevelMainTemplate = `// GENERATED CODE - DO NOT EDIT
package main
import (
"flag"
"reflect"
"github.com/revel/revel"{{range $k, $v := $.ImportPaths}}
{{$v}} "{{$k}}"{{end}}
"github.com/revel/revel/testing"
)
var (
runMode *string = flag.String("runMode", "", "Run mode.")
port *int = flag.Int("port", 0, "By default, read from app.conf")
importPath *string = flag.String("importPath", "", "Go Import Path for the app.")
srcPath *string = flag.String("srcPath", "", "Path to the source root.")
// So compiler won't complain if the generated code doesn't reference reflect package...
_ = reflect.Invalid
)
func main() {
flag.Parse()
revel.Init(*runMode, *importPath, *srcPath)
revel.AppLog.Info("Running revel server")
{{range $i, $c := .Controllers}}
revel.RegisterController((*{{index $.ImportPaths .ImportPath}}.{{.StructName}})(nil),
[]*revel.MethodType{
{{range .MethodSpecs}}&revel.MethodType{
Name: "{{.Name}}",
Args: []*revel.MethodArg{ {{range .Args}}
&revel.MethodArg{Name: "{{.Name}}", Type: reflect.TypeOf((*{{index $.ImportPaths .ImportPath | .TypeExpr.TypeName}})(nil)) },{{end}}
},
RenderArgNames: map[int][]string{ {{range .RenderCalls}}
{{.Line}}: []string{ {{range .Names}}
"{{.}}",{{end}}
},{{end}}
},
},
{{end}}
})
{{end}}
revel.DefaultValidationKeys = map[string]map[int]string{ {{range $path, $lines := .ValidationKeys}}
"{{$path}}": { {{range $line, $key := $lines}}
{{$line}}: "{{$key}}",{{end}}
},{{end}}
}
testing.TestSuites = []interface{}{ {{range .TestSuites}}
(*{{index $.ImportPaths .ImportPath}}.{{.StructName}})(nil),{{end}}
}
revel.Run(*port)
}
`
// RevelRoutesTemplate template for app/conf/routes
const RevelRoutesTemplate = `// GENERATED CODE - DO NOT EDIT
package routes
import "github.com/revel/revel"
{{range $i, $c := .Controllers}}
type t{{.StructName}} struct {}
var {{.StructName}} t{{.StructName}}
{{range .MethodSpecs}}
func (_ t{{$c.StructName}}) {{.Name}}({{range .Args}}
{{.Name}} {{if .ImportPath}}interface{}{{else}}{{.TypeExpr.TypeName ""}}{{end}},{{end}}
) string {
args := make(map[string]string)
{{range .Args}}
revel.Unbind(args, "{{.Name}}", {{.Name}}){{end}}
return revel.MainRouter.Reverse("{{$c.StructName}}.{{.Name}}", args).URL
}
{{end}}
{{end}}
`
+285
View File
@@ -0,0 +1,285 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
// Package harness for a Revel Framework.
//
// It has a following responsibilities:
// 1. Parse the user program, generating a main.go file that registers
// controller classes and starts the user's server.
// 2. Build and run the user program. Show compile errors.
// 3. Monitor the user source and re-build / restart the program when necessary.
//
// Source files are generated in the app/tmp directory.
package harness
import (
"crypto/tls"
"fmt"
"go/build"
"io"
"net"
"net/http"
"net/http/httputil"
"net/url"
"os"
"os/signal"
"path/filepath"
"strings"
"sync/atomic"
"github.com/revel/revel"
"sync"
)
var (
doNotWatch = []string{"tmp", "views", "routes"}
lastRequestHadError int32
)
// Harness reverse proxies requests to the application server.
// It builds / runs / rebuilds / restarts the server when code is changed.
type Harness struct {
app *App
serverHost string
port int
proxy *httputil.ReverseProxy
watcher *revel.Watcher
mutex *sync.Mutex
}
func renderError(iw http.ResponseWriter, ir *http.Request, err error) {
context := revel.NewGoContext(nil)
context.Request.SetRequest(ir)
context.Response.SetResponse(iw)
c := revel.NewController(context)
c.RenderError(err).Apply(c.Request, c.Response)
}
// ServeHTTP handles all requests.
// It checks for changes to app, rebuilds if necessary, and forwards the request.
func (h *Harness) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Don't rebuild the app for favicon requests.
if lastRequestHadError > 0 && r.URL.Path == "/favicon.ico" {
return
}
// Flush any change events and rebuild app if necessary.
// Render an error page if the rebuild / restart failed.
err := h.watcher.Notify()
if err != nil {
// In a thread safe manner update the flag so that a request for
// /favicon.ico does not trigger a rebuild
atomic.CompareAndSwapInt32(&lastRequestHadError, 0, 1)
renderError(w, r, err)
return
}
// In a thread safe manner update the flag so that a request for
// /favicon.ico is allowed
atomic.CompareAndSwapInt32(&lastRequestHadError, 1, 0)
// Reverse proxy the request.
// (Need special code for websockets, courtesy of bradfitz)
if strings.EqualFold(r.Header.Get("Upgrade"), "websocket") {
proxyWebsocket(w, r, h.serverHost)
} else {
h.proxy.ServeHTTP(w, r)
}
}
// NewHarness method returns a reverse proxy that forwards requests
// to the given port.
func NewHarness() *Harness {
// Get a template loader to render errors.
// Prefer the app's views/errors directory, and fall back to the stock error pages.
revel.MainTemplateLoader = revel.NewTemplateLoader(
[]string{filepath.Join(revel.RevelPath, "templates")})
if err := revel.MainTemplateLoader.Refresh(); err != nil {
revel.RevelLog.Error("Template loader error", "error", err)
}
addr := revel.HTTPAddr
port := revel.Config.IntDefault("harness.port", 0)
scheme := "http"
if revel.HTTPSsl {
scheme = "https"
}
// If the server is running on the wildcard address, use "localhost"
if addr == "" {
addr = "localhost"
}
if port == 0 {
port = getFreePort()
}
serverURL, _ := url.ParseRequestURI(fmt.Sprintf(scheme+"://%s:%d", addr, port))
serverHarness := &Harness{
port: port,
serverHost: serverURL.String()[len(scheme+"://"):],
proxy: httputil.NewSingleHostReverseProxy(serverURL),
mutex: &sync.Mutex{},
}
if revel.HTTPSsl {
serverHarness.proxy.Transport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
}
return serverHarness
}
// Refresh method rebuilds the Revel application and run it on the given port.
func (h *Harness) Refresh() (err *revel.Error) {
// Allow only one thread to rebuild the process
h.mutex.Lock()
defer h.mutex.Unlock()
if h.app != nil {
h.app.Kill()
}
revel.RevelLog.Debug("Rebuild Called")
h.app, err = Build()
if err != nil {
return
}
h.app.Port = h.port
if err2 := h.app.Cmd().Start(); err2 != nil {
return &revel.Error{
Title: "App failed to start up",
Description: err2.Error(),
}
}
return
}
// WatchDir method returns false to file matches with doNotWatch
// otheriwse true
func (h *Harness) WatchDir(info os.FileInfo) bool {
return !revel.ContainsString(doNotWatch, info.Name())
}
// WatchFile method returns true given filename HasSuffix of ".go"
// otheriwse false - implements revel.DiscerningListener
func (h *Harness) WatchFile(filename string) bool {
return strings.HasSuffix(filename, ".go")
}
// Run the harness, which listens for requests and proxies them to the app
// server, which it runs and rebuilds as necessary.
func (h *Harness) Run() {
var paths []string
if revel.Config.BoolDefault("watch.gopath", false) {
gopaths := filepath.SplitList(build.Default.GOPATH)
paths = append(paths, gopaths...)
}
paths = append(paths, revel.CodePaths...)
h.watcher = revel.NewWatcher()
h.watcher.Listen(h, paths...)
h.watcher.Notify()
go func() {
addr := fmt.Sprintf("%s:%d", revel.HTTPAddr, revel.HTTPPort)
revel.RevelLog.Infof("Listening on %s", addr)
var err error
if revel.HTTPSsl {
err = http.ListenAndServeTLS(
addr,
revel.HTTPSslCert,
revel.HTTPSslKey,
h)
} else {
err = http.ListenAndServe(addr, h)
}
if err != nil {
revel.RevelLog.Error("Failed to start reverse proxy:", "error", err)
}
}()
// Kill the app on signal.
ch := make(chan os.Signal)
signal.Notify(ch, os.Interrupt, os.Kill)
<-ch
if h.app != nil {
h.app.Kill()
}
os.Exit(1)
}
// Find an unused port
func getFreePort() (port int) {
conn, err := net.Listen("tcp", ":0")
if err != nil {
revel.RevelLog.Fatal("Unable to fetch a freee port address", "error", err)
}
port = conn.Addr().(*net.TCPAddr).Port
err = conn.Close()
if err != nil {
revel.RevelLog.Fatal("Unable to close port", "error", err)
}
return port
}
// proxyWebsocket copies data between websocket client and server until one side
// closes the connection. (ReverseProxy doesn't work with websocket requests.)
func proxyWebsocket(w http.ResponseWriter, r *http.Request, host string) {
var (
d net.Conn
err error
)
if revel.HTTPSsl {
// since this proxy isn't used in production,
// it's OK to set InsecureSkipVerify to true
// no need to add another configuration option.
d, err = tls.Dial("tcp", host, &tls.Config{InsecureSkipVerify: true})
} else {
d, err = net.Dial("tcp", host)
}
if err != nil {
http.Error(w, "Error contacting backend server.", 500)
revel.RevelLog.Error("Error dialing websocket backend ", "host", host, "error", err)
return
}
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "Not a hijacker?", 500)
return
}
nc, _, err := hj.Hijack()
if err != nil {
revel.RevelLog.Error("Hijack error", "error", err)
return
}
defer func() {
if err = nc.Close(); err != nil {
revel.RevelLog.Error("Connection close error", "error", err)
}
if err = d.Close(); err != nil {
revel.RevelLog.Error("Dial close error", "error", err)
}
}()
err = r.Write(d)
if err != nil {
revel.RevelLog.Error("Error copying request to target", "error", err)
return
}
errc := make(chan error, 2)
cp := func(dst io.Writer, src io.Reader) {
_, err := io.Copy(dst, src)
errc <- err
}
go cp(d, nc)
go cp(nc, d)
<-errc
}
+843
View File
@@ -0,0 +1,843 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package harness
// This file handles the app code introspection.
// It catalogs the controllers, their methods, and their arguments.
import (
"go/ast"
"go/build"
"go/parser"
"go/scanner"
"go/token"
"os"
"path/filepath"
"strings"
"github.com/revel/revel"
"unicode"
)
// SourceInfo is the top-level struct containing all extracted information
// about the app source code, used to generate main.go.
type SourceInfo struct {
// StructSpecs lists type info for all structs found under the code paths.
// They may be queried to determine which ones (transitively) embed certain types.
StructSpecs []*TypeInfo
// ValidationKeys provides a two-level lookup. The keys are:
// 1. The fully-qualified function name,
// e.g. "github.com/revel/examples/chat/app/controllers.(*Application).Action"
// 2. Within that func's file, the line number of the (overall) expression statement.
// e.g. the line returned from runtime.Caller()
// The result of the lookup the name of variable being validated.
ValidationKeys map[string]map[int]string
// A list of import paths.
// Revel notices files with an init() function and imports that package.
InitImportPaths []string
// controllerSpecs lists type info for all structs found under
// app/controllers/... that embed (directly or indirectly) revel.Controller
controllerSpecs []*TypeInfo
// testSuites list the types that constitute the set of application tests.
testSuites []*TypeInfo
}
// TypeInfo summarizes information about a struct type in the app source code.
type TypeInfo struct {
StructName string // e.g. "Application"
ImportPath string // e.g. "github.com/revel/examples/chat/app/controllers"
PackageName string // e.g. "controllers"
MethodSpecs []*MethodSpec
// Used internally to identify controllers that indirectly embed *revel.Controller.
embeddedTypes []*embeddedTypeName
}
// methodCall describes a call to c.Render(..)
// It documents the argument names used, in order to propagate them to RenderArgs.
type methodCall struct {
Path string // e.g. "myapp/app/controllers.(*Application).Action"
Line int
Names []string
}
// MethodSpec holds the information of one Method
type MethodSpec struct {
Name string // Name of the method, e.g. "Index"
Args []*MethodArg // Argument descriptors
RenderCalls []*methodCall // Descriptions of Render() invocations from this Method.
}
// MethodArg holds the information of one argument
type MethodArg struct {
Name string // Name of the argument.
TypeExpr TypeExpr // The name of the type, e.g. "int", "*pkg.UserType"
ImportPath string // If the arg is of an imported type, this is the import path.
}
type embeddedTypeName struct {
ImportPath, StructName string
}
// Maps a controller simple name (e.g. "Login") to the methods for which it is a
// receiver.
type methodMap map[string][]*MethodSpec
// ProcessSource parses the app controllers directory and
// returns a list of the controller types found.
// Otherwise CompileError if the parsing fails.
func ProcessSource(roots []string) (*SourceInfo, *revel.Error) {
var (
srcInfo *SourceInfo
compileError *revel.Error
)
for _, root := range roots {
rootImportPath := importPathFromPath(root)
if rootImportPath == "" {
revel.RevelLog.Warn("Skipping empty code path", "path", root)
continue
}
// Start walking the directory tree.
_ = revel.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
revel.RevelLog.Error("Error scanning app source:", "error", err)
return nil
}
if !info.IsDir() || info.Name() == "tmp" {
return nil
}
// Get the import path of the package.
pkgImportPath := rootImportPath
if root != path {
pkgImportPath = rootImportPath + "/" + filepath.ToSlash(path[len(root)+1:])
}
// Parse files within the path.
var pkgs map[string]*ast.Package
fset := token.NewFileSet()
pkgs, err = parser.ParseDir(fset, path, func(f os.FileInfo) bool {
return !f.IsDir() && !strings.HasPrefix(f.Name(), ".") && strings.HasSuffix(f.Name(), ".go")
}, 0)
if err != nil {
if errList, ok := err.(scanner.ErrorList); ok {
var pos = errList[0].Pos
compileError = &revel.Error{
SourceType: ".go source",
Title: "Go Compilation Error",
Path: pos.Filename,
Description: errList[0].Msg,
Line: pos.Line,
Column: pos.Column,
SourceLines: revel.MustReadLines(pos.Filename),
}
errorLink := revel.Config.StringDefault("error.link", "")
if errorLink != "" {
compileError.SetLink(errorLink)
}
return compileError
}
// This is exception, err alredy checked above. Here just a print
ast.Print(nil, err)
revel.RevelLog.Fatal("Failed to parse dir", "error", err)
}
// Skip "main" packages.
delete(pkgs, "main")
// If there is no code in this directory, skip it.
if len(pkgs) == 0 {
return nil
}
// Ignore packages that end with _test
for i := range pkgs {
if len(i) > 6 {
if string(i[len(i)-5:]) == "_test" {
delete(pkgs, i)
}
}
}
// There should be only one package in this directory.
if len(pkgs) > 1 {
for i := range pkgs {
println("Found package ", i)
}
revel.RevelLog.Error("Most unexpected! Multiple packages in a single directory:", "packages", pkgs)
}
var pkg *ast.Package
for _, v := range pkgs {
pkg = v
}
srcInfo = appendSourceInfo(srcInfo, processPackage(fset, pkgImportPath, path, pkg))
return nil
})
}
return srcInfo, compileError
}
func appendSourceInfo(srcInfo1, srcInfo2 *SourceInfo) *SourceInfo {
if srcInfo1 == nil {
return srcInfo2
}
srcInfo1.StructSpecs = append(srcInfo1.StructSpecs, srcInfo2.StructSpecs...)
srcInfo1.InitImportPaths = append(srcInfo1.InitImportPaths, srcInfo2.InitImportPaths...)
for k, v := range srcInfo2.ValidationKeys {
if _, ok := srcInfo1.ValidationKeys[k]; ok {
revel.RevelLog.Warn("Key conflict when scanning validation calls:", "key", k)
continue
}
srcInfo1.ValidationKeys[k] = v
}
return srcInfo1
}
func processPackage(fset *token.FileSet, pkgImportPath, pkgPath string, pkg *ast.Package) *SourceInfo {
var (
structSpecs []*TypeInfo
initImportPaths []string
methodSpecs = make(methodMap)
validationKeys = make(map[string]map[int]string)
scanControllers = strings.HasSuffix(pkgImportPath, "/controllers") ||
strings.Contains(pkgImportPath, "/controllers/")
scanTests = strings.HasSuffix(pkgImportPath, "/tests") ||
strings.Contains(pkgImportPath, "/tests/")
)
// For each source file in the package...
for _, file := range pkg.Files {
// Imports maps the package key to the full import path.
// e.g. import "sample/app/models" => "models": "sample/app/models"
imports := map[string]string{}
// For each declaration in the source file...
for _, decl := range file.Decls {
addImports(imports, decl, pkgPath)
if scanControllers {
// Match and add both structs and methods
structSpecs = appendStruct(structSpecs, pkgImportPath, pkg, decl, imports, fset)
appendAction(fset, methodSpecs, decl, pkgImportPath, pkg.Name, imports)
} else if scanTests {
structSpecs = appendStruct(structSpecs, pkgImportPath, pkg, decl, imports, fset)
}
// If this is a func...
if funcDecl, ok := decl.(*ast.FuncDecl); ok {
// Scan it for validation calls
lineKeys := getValidationKeys(fset, funcDecl, imports)
if len(lineKeys) > 0 {
validationKeys[pkgImportPath+"."+getFuncName(funcDecl)] = lineKeys
}
// Check if it's an init function.
if funcDecl.Name.Name == "init" {
initImportPaths = []string{pkgImportPath}
}
}
}
}
// Add the method specs to the struct specs.
for _, spec := range structSpecs {
spec.MethodSpecs = methodSpecs[spec.StructName]
}
return &SourceInfo{
StructSpecs: structSpecs,
ValidationKeys: validationKeys,
InitImportPaths: initImportPaths,
}
}
// getFuncName returns a name for this func or method declaration.
// e.g. "(*Application).SayHello" for a method, "SayHello" for a func.
func getFuncName(funcDecl *ast.FuncDecl) string {
prefix := ""
if funcDecl.Recv != nil {
recvType := funcDecl.Recv.List[0].Type
if recvStarType, ok := recvType.(*ast.StarExpr); ok {
prefix = "(*" + recvStarType.X.(*ast.Ident).Name + ")"
} else {
prefix = recvType.(*ast.Ident).Name
}
prefix += "."
}
return prefix + funcDecl.Name.Name
}
func addImports(imports map[string]string, decl ast.Decl, srcDir string) {
genDecl, ok := decl.(*ast.GenDecl)
if !ok {
return
}
if genDecl.Tok != token.IMPORT {
return
}
for _, spec := range genDecl.Specs {
importSpec := spec.(*ast.ImportSpec)
var pkgAlias string
if importSpec.Name != nil {
pkgAlias = importSpec.Name.Name
if pkgAlias == "_" {
continue
}
}
quotedPath := importSpec.Path.Value // e.g. "\"sample/app/models\""
fullPath := quotedPath[1 : len(quotedPath)-1] // Remove the quotes
// If the package was not aliased (common case), we have to import it
// to see what the package name is.
// TODO: Can improve performance here a lot:
// 1. Do not import everything over and over again. Keep a cache.
// 2. Exempt the standard library; their directories always match the package name.
// 3. Can use build.FindOnly and then use parser.ParseDir with mode PackageClauseOnly
if pkgAlias == "" {
pkg, err := build.Import(fullPath, srcDir, 0)
if err != nil {
// We expect this to happen for apps using reverse routing (since we
// have not yet generated the routes). Don't log that.
if !strings.HasSuffix(fullPath, "/app/routes") {
revel.RevelLog.Debug("Could not find import:", "path", fullPath)
}
continue
}
pkgAlias = pkg.Name
}
imports[pkgAlias] = fullPath
}
}
// If this Decl is a struct type definition, it is summarized and added to specs.
// Else, specs is returned unchanged.
func appendStruct(specs []*TypeInfo, pkgImportPath string, pkg *ast.Package, decl ast.Decl, imports map[string]string, fset *token.FileSet) []*TypeInfo {
// Filter out non-Struct type declarations.
spec, found := getStructTypeDecl(decl, fset)
if !found {
return specs
}
structType := spec.Type.(*ast.StructType)
// At this point we know it's a type declaration for a struct.
// Fill in the rest of the info by diving into the fields.
// Add it provisionally to the Controller list -- it's later filtered using field info.
controllerSpec := &TypeInfo{
StructName: spec.Name.Name,
ImportPath: pkgImportPath,
PackageName: pkg.Name,
}
for _, field := range structType.Fields.List {
// If field.Names is set, it's not an embedded type.
if field.Names != nil {
continue
}
// A direct "sub-type" has an ast.Field as either:
// Ident { "AppController" }
// SelectorExpr { "rev", "Controller" }
// Additionally, that can be wrapped by StarExprs.
fieldType := field.Type
pkgName, typeName := func() (string, string) {
// Drill through any StarExprs.
for {
if starExpr, ok := fieldType.(*ast.StarExpr); ok {
fieldType = starExpr.X
continue
}
break
}
// If the embedded type is in the same package, it's an Ident.
if ident, ok := fieldType.(*ast.Ident); ok {
return "", ident.Name
}
if selectorExpr, ok := fieldType.(*ast.SelectorExpr); ok {
if pkgIdent, ok := selectorExpr.X.(*ast.Ident); ok {
return pkgIdent.Name, selectorExpr.Sel.Name
}
}
return "", ""
}()
// If a typename wasn't found, skip it.
if typeName == "" {
continue
}
// Find the import path for this type.
// If it was referenced without a package name, use the current package import path.
// Else, look up the package's import path by name.
var importPath string
if pkgName == "" {
importPath = pkgImportPath
} else {
var ok bool
if importPath, ok = imports[pkgName]; !ok {
revel.RevelLog.Error("Failed to find import path for ", "package", pkgName, "type", typeName)
continue
}
}
controllerSpec.embeddedTypes = append(controllerSpec.embeddedTypes, &embeddedTypeName{
ImportPath: importPath,
StructName: typeName,
})
}
return append(specs, controllerSpec)
}
// If decl is a Method declaration, it is summarized and added to the array
// underneath its receiver type.
// e.g. "Login" => {MethodSpec, MethodSpec, ..}
func appendAction(fset *token.FileSet, mm methodMap, decl ast.Decl, pkgImportPath, pkgName string, imports map[string]string) {
// Func declaration?
funcDecl, ok := decl.(*ast.FuncDecl)
if !ok {
return
}
// Have a receiver?
if funcDecl.Recv == nil {
return
}
// Is it public?
if !funcDecl.Name.IsExported() {
return
}
// Does it return a Result?
if funcDecl.Type.Results == nil || len(funcDecl.Type.Results.List) != 1 {
return
}
selExpr, ok := funcDecl.Type.Results.List[0].Type.(*ast.SelectorExpr)
if !ok {
return
}
if selExpr.Sel.Name != "Result" {
return
}
if pkgIdent, ok := selExpr.X.(*ast.Ident); !ok || imports[pkgIdent.Name] != revel.RevelImportPath {
return
}
method := &MethodSpec{
Name: funcDecl.Name.Name,
}
// Add a description of the arguments to the method.
for _, field := range funcDecl.Type.Params.List {
for _, name := range field.Names {
var importPath string
typeExpr := NewTypeExpr(pkgName, field.Type)
if !typeExpr.Valid {
revel.RevelLog.Warnf("Didn't understand argument '%s' of action %s. Ignoring.", name, getFuncName(funcDecl))
return // We didn't understand one of the args. Ignore this action.
}
// Local object
if typeExpr.PkgName == pkgName {
importPath = pkgImportPath
} else if typeExpr.PkgName != "" {
var ok bool
if importPath, ok = imports[typeExpr.PkgName]; !ok {
revel.RevelLog.Errorf("Failed to find import for arg of type: %s , %s", typeExpr.PkgName, typeExpr.TypeName(""))
}
}
method.Args = append(method.Args, &MethodArg{
Name: name.Name,
TypeExpr: typeExpr,
ImportPath: importPath,
})
}
}
// Add a description of the calls to Render from the method.
// Inspect every node (e.g. always return true).
method.RenderCalls = []*methodCall{}
ast.Inspect(funcDecl.Body, func(node ast.Node) bool {
// Is it a function call?
callExpr, ok := node.(*ast.CallExpr)
if !ok {
return true
}
// Is it calling (*Controller).Render?
selExpr, ok := callExpr.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
// The type of the receiver is not easily available, so just store every
// call to any method called Render.
if selExpr.Sel.Name != "Render" {
return true
}
// Add this call's args to the renderArgs.
pos := fset.Position(callExpr.Lparen)
methodCall := &methodCall{
Line: pos.Line,
Names: []string{},
}
for _, arg := range callExpr.Args {
argIdent, ok := arg.(*ast.Ident)
if !ok {
continue
}
methodCall.Names = append(methodCall.Names, argIdent.Name)
}
method.RenderCalls = append(method.RenderCalls, methodCall)
return true
})
var recvTypeName string
var recvType = funcDecl.Recv.List[0].Type
if recvStarType, ok := recvType.(*ast.StarExpr); ok {
recvTypeName = recvStarType.X.(*ast.Ident).Name
} else {
recvTypeName = recvType.(*ast.Ident).Name
}
mm[recvTypeName] = append(mm[recvTypeName], method)
}
// Scan app source code for calls to X.Y(), where X is of type *Validation.
//
// Recognize these scenarios:
// - "Y" = "Validation" and is a member of the receiver.
// (The common case for inline validation)
// - "X" is passed in to the func as a parameter.
// (For structs implementing Validated)
//
// The line number to which a validation call is attributed is that of the
// surrounding ExprStmt. This is so that it matches what runtime.Callers()
// reports.
//
// The end result is that we can set the default validation key for each call to
// be the same as the local variable.
func getValidationKeys(fset *token.FileSet, funcDecl *ast.FuncDecl, imports map[string]string) map[int]string {
var (
lineKeys = make(map[int]string)
// Check the func parameters and the receiver's members for the *revel.Validation type.
validationParam = getValidationParameter(funcDecl, imports)
)
ast.Inspect(funcDecl.Body, func(node ast.Node) bool {
// e.g. c.Validation.Required(arg) or v.Required(arg)
callExpr, ok := node.(*ast.CallExpr)
if !ok {
return true
}
// e.g. c.Validation.Required or v.Required
funcSelector, ok := callExpr.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch x := funcSelector.X.(type) {
case *ast.SelectorExpr: // e.g. c.Validation
if x.Sel.Name != "Validation" {
return true
}
case *ast.Ident: // e.g. v
if validationParam == nil || x.Obj != validationParam {
return true
}
default:
return true
}
if len(callExpr.Args) == 0 {
return true
}
// Given the validation expression, extract the key.
key := callExpr.Args[0]
switch expr := key.(type) {
case *ast.BinaryExpr:
// If the argument is a binary expression, take the first expression.
// (e.g. c.Validation.Required(myName != ""))
key = expr.X
case *ast.UnaryExpr:
// If the argument is a unary expression, drill in.
// (e.g. c.Validation.Required(!myBool)
key = expr.X
case *ast.BasicLit:
// If it's a literal, skip it.
return true
}
if typeExpr := NewTypeExpr("", key); typeExpr.Valid {
lineKeys[fset.Position(callExpr.End()).Line] = typeExpr.TypeName("")
}
return true
})
return lineKeys
}
// Check to see if there is a *revel.Validation as an argument.
func getValidationParameter(funcDecl *ast.FuncDecl, imports map[string]string) *ast.Object {
for _, field := range funcDecl.Type.Params.List {
starExpr, ok := field.Type.(*ast.StarExpr) // e.g. *revel.Validation
if !ok {
continue
}
selExpr, ok := starExpr.X.(*ast.SelectorExpr) // e.g. revel.Validation
if !ok {
continue
}
xIdent, ok := selExpr.X.(*ast.Ident) // e.g. rev
if !ok {
continue
}
if selExpr.Sel.Name == "Validation" && imports[xIdent.Name] == revel.RevelImportPath {
return field.Names[0].Obj
}
}
return nil
}
func (s *TypeInfo) String() string {
return s.ImportPath + "." + s.StructName
}
func (s *embeddedTypeName) String() string {
return s.ImportPath + "." + s.StructName
}
// getStructTypeDecl checks if the given decl is a type declaration for a
// struct. If so, the TypeSpec is returned.
func getStructTypeDecl(decl ast.Decl, fset *token.FileSet) (spec *ast.TypeSpec, found bool) {
genDecl, ok := decl.(*ast.GenDecl)
if !ok {
return
}
if genDecl.Tok != token.TYPE {
return
}
if len(genDecl.Specs) == 0 {
revel.RevelLog.Warnf("Surprising: %s:%d Decl contains no specifications", fset.Position(decl.Pos()).Filename, fset.Position(decl.Pos()).Line)
return
}
spec = genDecl.Specs[0].(*ast.TypeSpec)
_, found = spec.Type.(*ast.StructType)
return
}
// TypesThatEmbed returns all types that (directly or indirectly) embed the
// target type, which must be a fully qualified type name,
// e.g. "github.com/revel/revel.Controller"
func (s *SourceInfo) TypesThatEmbed(targetType, packageFilter string) (filtered []*TypeInfo) {
// Do a search in the "embedded type graph", starting with the target type.
var (
nodeQueue = []string{targetType}
processed []string
)
for len(nodeQueue) > 0 {
controllerSimpleName := nodeQueue[0]
nodeQueue = nodeQueue[1:]
processed = append(processed, controllerSimpleName)
// Look through all known structs.
for _, spec := range s.StructSpecs {
// If this one has been processed or is already in nodeQueue, then skip it.
if revel.ContainsString(processed, spec.String()) ||
revel.ContainsString(nodeQueue, spec.String()) {
continue
}
// Look through the embedded types to see if the current type is among them.
for _, embeddedType := range spec.embeddedTypes {
// If so, add this type's simple name to the nodeQueue, and its spec to
// the filtered list.
if controllerSimpleName == embeddedType.String() {
nodeQueue = append(nodeQueue, spec.String())
filtered = append(filtered, spec)
break
}
}
}
}
// Strip out any specifications that contain a lower case
for exit := false; !exit; exit = true {
for i, filteredItem := range filtered {
if unicode.IsLower([]rune(filteredItem.StructName)[0]) {
revel.RevelLog.Debug("Skipping adding spec for unexported type",
"type", filteredItem.StructName,
"package", filteredItem.ImportPath)
filtered = append(filtered[:i], filtered[i+1:]...)
exit = false
break
}
}
}
// Check for any missed types that where from expected packages
for _, spec := range s.StructSpecs {
if spec.PackageName == packageFilter {
found := false
for _, filteredItem := range filtered {
if filteredItem.StructName == spec.StructName {
found = true
break
}
}
if !found {
revel.RevelLog.Warn("Type found in package: "+packageFilter+
", but did not embed from: "+filepath.Base(targetType),
"name", spec.StructName, "path", spec.ImportPath)
}
}
}
return
}
// ControllerSpecs returns the all the contollers that embeds
// `revel.Controller`
func (s *SourceInfo) ControllerSpecs() []*TypeInfo {
if s.controllerSpecs == nil {
s.controllerSpecs = s.TypesThatEmbed(revel.RevelImportPath+".Controller", "controllers")
}
return s.controllerSpecs
}
// TestSuites returns the all the Application tests that embeds
// `testing.TestSuite`
func (s *SourceInfo) TestSuites() []*TypeInfo {
if s.testSuites == nil {
s.testSuites = s.TypesThatEmbed(revel.RevelImportPath+"/testing.TestSuite", "testsuite")
}
return s.testSuites
}
// TypeExpr provides a type name that may be rewritten to use a package name.
type TypeExpr struct {
Expr string // The unqualified type expression, e.g. "[]*MyType"
PkgName string // The default package idenifier
pkgIndex int // The index where the package identifier should be inserted.
Valid bool
}
// TypeName returns the fully-qualified type name for this expression.
// The caller may optionally specify a package name to override the default.
func (e TypeExpr) TypeName(pkgOverride string) string {
pkgName := revel.FirstNonEmpty(pkgOverride, e.PkgName)
if pkgName == "" {
return e.Expr
}
return e.Expr[:e.pkgIndex] + pkgName + "." + e.Expr[e.pkgIndex:]
}
// NewTypeExpr returns the syntactic expression for referencing this type in Go.
func NewTypeExpr(pkgName string, expr ast.Expr) TypeExpr {
switch t := expr.(type) {
case *ast.Ident:
if IsBuiltinType(t.Name) {
pkgName = ""
}
return TypeExpr{t.Name, pkgName, 0, true}
case *ast.SelectorExpr:
e := NewTypeExpr(pkgName, t.X)
return TypeExpr{t.Sel.Name, e.Expr, 0, e.Valid}
case *ast.StarExpr:
e := NewTypeExpr(pkgName, t.X)
return TypeExpr{"*" + e.Expr, e.PkgName, e.pkgIndex + 1, e.Valid}
case *ast.ArrayType:
e := NewTypeExpr(pkgName, t.Elt)
return TypeExpr{"[]" + e.Expr, e.PkgName, e.pkgIndex + 2, e.Valid}
case *ast.Ellipsis:
e := NewTypeExpr(pkgName, t.Elt)
return TypeExpr{"[]" + e.Expr, e.PkgName, e.pkgIndex + 2, e.Valid}
default:
revel.RevelLog.Error("Failed to generate name for field. Make sure the field name is valid.")
}
return TypeExpr{Valid: false}
}
var builtInTypes = map[string]struct{}{
"bool": {},
"byte": {},
"complex128": {},
"complex64": {},
"error": {},
"float32": {},
"float64": {},
"int": {},
"int16": {},
"int32": {},
"int64": {},
"int8": {},
"rune": {},
"string": {},
"uint": {},
"uint16": {},
"uint32": {},
"uint64": {},
"uint8": {},
"uintptr": {},
}
// IsBuiltinType checks the given type is built-in types of Go
func IsBuiltinType(name string) bool {
_, ok := builtInTypes[name]
return ok
}
func importPathFromPath(root string) string {
vendoringPath := revel.BasePath + "/vendor/"
if strings.HasPrefix(root, vendoringPath) {
return filepath.ToSlash(root[len(vendoringPath):])
}
for _, gopath := range filepath.SplitList(build.Default.GOPATH) {
srcPath := filepath.Join(gopath, "src")
if strings.HasPrefix(root, srcPath) {
return filepath.ToSlash(root[len(srcPath)+1:])
}
}
srcPath := filepath.Join(build.Default.GOROOT, "src", "pkg")
if strings.HasPrefix(root, srcPath) {
revel.RevelLog.Warn("Code path should be in GOPATH, but is in GOROOT:", "path", root)
return filepath.ToSlash(root[len(srcPath)+1:])
}
revel.RevelLog.Error("Unexpected! Code path is not in GOPATH:", "path", root)
return ""
}
+193
View File
@@ -0,0 +1,193 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package harness
import (
"go/ast"
"go/parser"
"go/token"
"reflect"
"strings"
"testing"
"github.com/revel/revel"
)
const validationKeysSource = `
package test
func (c *Application) testFunc(a, b int, user models.User) revel.Result {
// Line 5
c.Validation.Required(a)
c.Validation.Required(a).Message("Error message")
c.Validation.Required(a).
Message("Error message")
// Line 11
c.Validation.Required(user.Name)
c.Validation.Required(user.Name).Message("Error message")
// Line 15
c.Validation.MinSize(b, 12)
c.Validation.MinSize(b, 12).Message("Error message")
c.Validation.MinSize(b,
12)
// Line 21
c.Validation.Required(b == 5)
}
func (m Model) Validate(v *revel.Validation) {
// Line 26
v.Required(m.name)
v.Required(m.name == "something").
Message("Error Message")
v.Required(!m.bool)
}
`
var expectedValidationKeys = []map[int]string{
{
6: "a",
7: "a",
8: "a",
12: "user.Name",
13: "user.Name",
16: "b",
17: "b",
19: "b",
22: "b",
}, {
27: "m.name",
28: "m.name",
30: "m.bool",
},
}
// This tests the recording of line number to validation key of the preceeding
// example source.
func TestGetValidationKeys(t *testing.T) {
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, "validationKeysSource", validationKeysSource, 0)
if err != nil {
t.Fatal(err)
}
if len(file.Decls) != 2 {
t.Fatal("Expected 2 decl in the source, found", len(file.Decls))
}
for i, decl := range file.Decls {
lineKeys := getValidationKeys(fset, decl.(*ast.FuncDecl), map[string]string{"revel": revel.RevelImportPath})
for k, v := range expectedValidationKeys[i] {
if lineKeys[k] != v {
t.Errorf("Not found - %d: %v - Actual Map: %v", k, v, lineKeys)
}
}
if len(lineKeys) != len(expectedValidationKeys[i]) {
t.Error("Validation key map not the same size as expected:", lineKeys)
}
}
}
var TypeExprs = map[string]TypeExpr{
"int": {"int", "", 0, true},
"*int": {"*int", "", 1, true},
"[]int": {"[]int", "", 2, true},
"...int": {"[]int", "", 2, true},
"[]*int": {"[]*int", "", 3, true},
"...*int": {"[]*int", "", 3, true},
"MyType": {"MyType", "pkg", 0, true},
"*MyType": {"*MyType", "pkg", 1, true},
"[]MyType": {"[]MyType", "pkg", 2, true},
"...MyType": {"[]MyType", "pkg", 2, true},
"[]*MyType": {"[]*MyType", "pkg", 3, true},
"...*MyType": {"[]*MyType", "pkg", 3, true},
}
func TestTypeExpr(t *testing.T) {
for typeStr, expected := range TypeExprs {
// Handle arrays and ... myself, since ParseExpr() does not.
array := strings.HasPrefix(typeStr, "[]")
if array {
typeStr = typeStr[2:]
}
ellipsis := strings.HasPrefix(typeStr, "...")
if ellipsis {
typeStr = typeStr[3:]
}
expr, err := parser.ParseExpr(typeStr)
if err != nil {
t.Error("Failed to parse test expr:", typeStr)
continue
}
if array {
expr = &ast.ArrayType{Lbrack: expr.Pos(), Len: nil, Elt: expr}
}
if ellipsis {
expr = &ast.Ellipsis{Ellipsis: expr.Pos(), Elt: expr}
}
actual := NewTypeExpr("pkg", expr)
if !reflect.DeepEqual(expected, actual) {
t.Error("Fail, expected", expected, ", was", actual)
}
}
}
func TestProcessBookingSource(t *testing.T) {
revel.Init("prod", "github.com/revel/examples/booking", "")
sourceInfo, err := ProcessSource([]string{revel.AppPath})
if err != nil {
t.Fatal("Failed to process booking source with error:", err)
}
controllerPackage := "github.com/revel/examples/booking/app/controllers"
expectedControllerSpecs := []*TypeInfo{
{"GorpController", controllerPackage, "controllers", nil, nil},
{"Application", controllerPackage, "controllers", nil, nil},
{"Hotels", controllerPackage, "controllers", nil, nil},
}
if len(sourceInfo.ControllerSpecs()) != len(expectedControllerSpecs) {
t.Errorf("Unexpected number of controllers found. Expected %d, Found %d",
len(expectedControllerSpecs), len(sourceInfo.ControllerSpecs()))
}
NEXT_TEST:
for _, expected := range expectedControllerSpecs {
for _, actual := range sourceInfo.ControllerSpecs() {
if actual.StructName == expected.StructName {
if actual.ImportPath != expected.ImportPath {
t.Errorf("%s expected to have import path %s, actual %s",
actual.StructName, expected.ImportPath, actual.ImportPath)
}
if actual.PackageName != expected.PackageName {
t.Errorf("%s expected to have package name %s, actual %s",
actual.StructName, expected.PackageName, actual.PackageName)
}
continue NEXT_TEST
}
}
t.Errorf("Expected to find controller %s, but did not. Actuals: %s",
expected.StructName, sourceInfo.ControllerSpecs())
}
}
func BenchmarkProcessBookingSource(b *testing.B) {
revel.Init("", "github.com/revel/examples/booking", "")
revel.GetRootLogHandler().Disable()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := ProcessSource(revel.CodePaths)
if err != nil {
b.Error("Unexpected error:", err)
}
}
}
+130
View File
@@ -0,0 +1,130 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"github.com/revel/cmd/harness"
"github.com/revel/revel"
)
var cmdBuild = &Command{
UsageLine: "build [import path] [target path] [run mode]",
Short: "build a Revel application (e.g. for deployment)",
Long: `
Build the Revel web application named by the given import path.
This allows it to be deployed and run on a machine that lacks a Go installation.
The run mode is used to select which set of app.conf configuration should
apply and may be used to determine logic in the application itself.
Run mode defaults to "dev".
WARNING: The target path will be completely deleted, if it already exists!
For example:
revel build github.com/revel/examples/chat /tmp/chat
`,
}
func init() {
cmdBuild.Run = buildApp
}
func buildApp(args []string) {
if len(args) < 2 {
fmt.Fprintf(os.Stderr, "%s\n%s", cmdBuild.UsageLine, cmdBuild.Long)
return
}
appImportPath, destPath, mode := args[0], args[1], DefaultRunMode
if len(args) >= 3 {
mode = args[2]
}
if !revel.Initialized {
revel.Init(mode, appImportPath, "")
}
// First, verify that it is either already empty or looks like a previous
// build (to avoid clobbering anything)
if exists(destPath) && !empty(destPath) && !exists(filepath.Join(destPath, "run.sh")) {
errorf("Abort: %s exists and does not look like a build directory.", destPath)
}
if err := os.RemoveAll(destPath); err != nil && !os.IsNotExist(err) {
revel.RevelLog.Fatal("Remove all error","error", err)
}
if err := os.MkdirAll(destPath, 0777); err != nil {
revel.RevelLog.Fatal("makedir error","error",err)
}
app, reverr := harness.Build()
panicOnError(reverr, "Failed to build")
// Included are:
// - run scripts
// - binary
// - revel
// - app
// Revel and the app are in a directory structure mirroring import path
srcPath := filepath.Join(destPath, "src")
destBinaryPath := filepath.Join(destPath, filepath.Base(app.BinaryPath))
tmpRevelPath := filepath.Join(srcPath, filepath.FromSlash(revel.RevelImportPath))
mustCopyFile(destBinaryPath, app.BinaryPath)
mustChmod(destBinaryPath, 0755)
_ = mustCopyDir(filepath.Join(tmpRevelPath, "conf"), filepath.Join(revel.RevelPath, "conf"), nil)
_ = mustCopyDir(filepath.Join(tmpRevelPath, "templates"), filepath.Join(revel.RevelPath, "templates"), nil)
_ = mustCopyDir(filepath.Join(srcPath, filepath.FromSlash(appImportPath)), revel.BasePath, nil)
// Find all the modules used and copy them over.
config := revel.Config.Raw()
modulePaths := make(map[string]string) // import path => filesystem path
for _, section := range config.Sections() {
options, _ := config.SectionOptions(section)
for _, key := range options {
if !strings.HasPrefix(key, "module.") {
continue
}
moduleImportPath, _ := config.String(section, key)
if moduleImportPath == "" {
continue
}
modulePath, err := revel.ResolveImportPath(moduleImportPath)
if err != nil {
revel.RevelLog.Fatalf("Failed to load module %s: %s", key[len("module."):], err)
}
modulePaths[moduleImportPath] = modulePath
}
}
for importPath, fsPath := range modulePaths {
_ = mustCopyDir(filepath.Join(srcPath, importPath), fsPath, nil)
}
tmplData, runShPath := map[string]interface{}{
"BinName": filepath.Base(app.BinaryPath),
"ImportPath": appImportPath,
"Mode": mode,
}, filepath.Join(destPath, "run.sh")
mustRenderTemplate(
runShPath,
filepath.Join(revel.RevelPath, "..", "cmd", "revel", "package_run.sh.template"),
tmplData)
mustChmod(runShPath, 0755)
mustRenderTemplate(
filepath.Join(destPath, "run.bat"),
filepath.Join(revel.RevelPath, "..", "cmd", "revel", "package_run.bat.template"),
tmplData)
}
+57
View File
@@ -0,0 +1,57 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package main
import (
"fmt"
"go/build"
"os"
"path/filepath"
)
var cmdClean = &Command{
UsageLine: "clean [import path]",
Short: "clean a Revel application's temp files",
Long: `
Clean the Revel web application named by the given import path.
For example:
revel clean github.com/revel/examples/chat
It removes the app/tmp and app/routes directory.
`,
}
func init() {
cmdClean.Run = cleanApp
}
func cleanApp(args []string) {
if len(args) == 0 {
fmt.Fprintf(os.Stderr, cmdClean.Long)
return
}
appPkg, err := build.Import(args[0], "", build.FindOnly)
if err != nil {
fmt.Fprintln(os.Stderr, "Abort: Failed to find import path:", err)
return
}
purgeDirs := []string{
filepath.Join(appPkg.Dir, "app", "tmp"),
filepath.Join(appPkg.Dir, "app", "routes"),
}
for _, dir := range purgeDirs {
fmt.Println("Removing:", dir)
err = os.RemoveAll(dir)
if err != nil {
fmt.Fprintln(os.Stderr, "Abort:", err)
return
}
}
}
+221
View File
@@ -0,0 +1,221 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package main
import (
"bytes"
"fmt"
"go/build"
"math/rand"
"os"
"os/exec"
"path/filepath"
"strings"
"github.com/revel/revel"
)
var cmdNew = &Command{
UsageLine: "new [path] [skeleton]",
Short: "create a skeleton Revel application",
Long: `
New creates a few files to get a new Revel application running quickly.
It puts all of the files in the given import path, taking the final element in
the path to be the app name.
Skeleton is an optional argument, provided as an import path
For example:
revel new import/path/helloworld
revel new import/path/helloworld import/path/skeleton
`,
}
func init() {
cmdNew.Run = newApp
}
var (
// go related paths
gopath string
gocmd string
srcRoot string
// revel related paths
revelPkg *build.Package
revelCmdPkg *build.Package
appPath string
appName string
basePath string
importPath string
skeletonPath string
)
func newApp(args []string) {
// check for proper args by count
if len(args) == 0 {
errorf("No import path given.\nRun 'revel help new' for usage.\n")
}
if len(args) > 2 {
errorf("Too many arguments provided.\nRun 'revel help new' for usage.\n")
}
// checking and setting go paths
initGoPaths()
// checking and setting application
setApplicationPath(args)
// checking and setting skeleton
setSkeletonPath(args)
// copy files to new app directory
copyNewAppFiles()
// goodbye world
fmt.Fprintln(os.Stdout, "Your application is ready:\n ", appPath)
fmt.Fprintln(os.Stdout, "\nYou can run it with:\n revel run", importPath)
}
const alphaNumeric = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
func generateSecret() string {
chars := make([]byte, 64)
for i := 0; i < 64; i++ {
chars[i] = alphaNumeric[rand.Intn(len(alphaNumeric))]
}
return string(chars)
}
// lookup and set Go related variables
func initGoPaths() {
// lookup go path
gopath = build.Default.GOPATH
if gopath == "" {
errorf("Abort: GOPATH environment variable is not set. " +
"Please refer to http://golang.org/doc/code.html to configure your Go environment.")
}
// check for go executable
var err error
gocmd, err = exec.LookPath("go")
if err != nil {
errorf("Go executable not found in PATH.")
}
// revel/revel#1004 choose go path relative to current working directory
workingDir, _ := os.Getwd()
goPathList := filepath.SplitList(gopath)
for _, path := range goPathList {
if strings.HasPrefix(strings.ToLower(workingDir), strings.ToLower(path)) {
srcRoot = path
break
}
path, _ = filepath.EvalSymlinks(path)
if len(path) > 0 && strings.HasPrefix(strings.ToLower(workingDir), strings.ToLower(path)) {
srcRoot = path
break
}
}
if len(srcRoot) == 0 {
revel.RevelLog.Fatal("Abort: could not create a Revel application outside of GOPATH.")
}
// set go src path
srcRoot = filepath.Join(srcRoot, "src")
}
func setApplicationPath(args []string) {
var err error
importPath = args[0]
// revel/revel#1014 validate relative path, we cannot use built-in functions
// since Go import path is valid relative path too.
// so check basic part of the path, which is "."
if filepath.IsAbs(importPath) || strings.HasPrefix(importPath, ".") {
errorf("Abort: '%s' looks like a directory. Please provide a Go import path instead.",
importPath)
}
appPath = filepath.Join(srcRoot, filepath.FromSlash(importPath))
_, err = build.Import(importPath, "", build.FindOnly)
if err == nil && !empty(appPath) {
errorf("Abort: Import path %s already exists.\n", importPath)
}
revelPkg, err = build.Import(revel.RevelImportPath, "", build.FindOnly)
if err != nil {
errorf("Abort: Could not find Revel source code: %s\n", err)
}
appName = filepath.Base(appPath)
basePath = filepath.ToSlash(filepath.Dir(importPath))
if basePath == "." {
// we need to remove the a single '.' when
// the app is in the $GOROOT/src directory
basePath = ""
} else {
// we need to append a '/' when the app is
// is a subdirectory such as $GOROOT/src/path/to/revelapp
basePath += "/"
}
}
func setSkeletonPath(args []string) {
var err error
if len(args) == 2 { // user specified
skeletonName := args[1]
_, err = build.Import(skeletonName, "", build.FindOnly)
if err != nil {
// Execute "go get <pkg>"
getCmd := exec.Command(gocmd, "get", "-d", skeletonName)
fmt.Println("Exec:", getCmd.Args)
getOutput, err := getCmd.CombinedOutput()
// check getOutput for no buildible string
bpos := bytes.Index(getOutput, []byte("no buildable Go source files in"))
if err != nil && bpos == -1 {
errorf("Abort: Could not find or 'go get' Skeleton source code: %s\n%s\n", getOutput, skeletonName)
}
}
// use the
skeletonPath = filepath.Join(srcRoot, skeletonName)
} else {
// use the revel default
revelCmdPkg, err = build.Import(RevelCmdImportPath, "", build.FindOnly)
if err != nil {
errorf("Abort: Could not find Revel Cmd source code: %s\n", err)
}
skeletonPath = filepath.Join(revelCmdPkg.Dir, "revel", "skeleton")
}
}
func copyNewAppFiles() {
var err error
err = os.MkdirAll(appPath, 0777)
panicOnError(err, "Failed to create directory "+appPath)
_ = mustCopyDir(appPath, skeletonPath, map[string]interface{}{
// app.conf
"AppName": appName,
"BasePath": basePath,
"Secret": generateSecret(),
})
// Dotfiles are skipped by mustCopyDir, so we have to explicitly copy the .gitignore.
gitignore := ".gitignore"
mustCopyFile(filepath.Join(appPath, gitignore), filepath.Join(skeletonPath, gitignore))
}
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/revel/revel"
)
var cmdPackage = &Command{
UsageLine: "package [import path] [run mode]",
Short: "package a Revel application (e.g. for deployment)",
Long: `
Package the Revel web application named by the given import path.
This allows it to be deployed and run on a machine that lacks a Go installation.
The run mode is used to select which set of app.conf configuration should
apply and may be used to determine logic in the application itself.
Run mode defaults to "dev".
For example:
revel package github.com/revel/examples/chat
`,
}
func init() {
cmdPackage.Run = packageApp
}
func packageApp(args []string) {
if len(args) == 0 {
fmt.Fprint(os.Stderr, cmdPackage.Long)
return
}
// Determine the run mode.
mode := DefaultRunMode
if len(args) >= 2 {
mode = args[1]
}
appImportPath := args[0]
revel.Init(mode, appImportPath, "")
// Remove the archive if it already exists.
destFile := filepath.Base(revel.BasePath) + ".tar.gz"
if err := os.Remove(destFile); err != nil && !os.IsNotExist(err) {
revel.RevelLog.Fatal("Unable to remove target file","error",err,"file",destFile)
}
// Collect stuff in a temp directory.
tmpDir, err := ioutil.TempDir("", filepath.Base(revel.BasePath))
panicOnError(err, "Failed to get temp dir")
buildApp([]string{args[0], tmpDir, mode})
// Create the zip file.
archiveName := mustTarGzDir(destFile, tmpDir)
fmt.Println("Your archive is ready:", archiveName)
}
+2
View File
@@ -0,0 +1,2 @@
@echo off
{{.BinName}} -importPath {{.ImportPath}} -srcPath %CD%\src -runMode {{.Mode}}
+3
View File
@@ -0,0 +1,3 @@
#!/bin/sh
SCRIPTPATH=$(cd "$(dirname "$0")"; pwd)
"$SCRIPTPATH/{{.BinName}}" -importPath {{.ImportPath}} -srcPath "$SCRIPTPATH/src" -runMode {{.Mode}}
+144
View File
@@ -0,0 +1,144 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
// The command line tool for running Revel apps.
package main
import (
"flag"
"fmt"
"io"
"math/rand"
"os"
"runtime"
"strings"
"text/template"
"time"
"github.com/agtorre/gocolorize"
)
const (
// RevelCmdImportPath Revel framework cmd tool import path
RevelCmdImportPath = "github.com/revel/cmd"
// DefaultRunMode for revel's application
DefaultRunMode = "dev"
)
// Command structure cribbed from the genius organization of the "go" command.
type Command struct {
Run func(args []string)
UsageLine, Short, Long string
}
// Name returns command name from usage line
func (cmd *Command) Name() string {
name := cmd.UsageLine
i := strings.Index(name, " ")
if i >= 0 {
name = name[:i]
}
return name
}
var commands = []*Command{
cmdNew,
cmdRun,
cmdBuild,
cmdPackage,
cmdClean,
cmdTest,
cmdVersion,
}
func main() {
if runtime.GOOS == "windows" {
gocolorize.SetPlain(true)
}
fmt.Fprintf(os.Stdout, gocolorize.NewColor("blue").Paint(header))
flag.Usage = func() { usage(1) }
flag.Parse()
args := flag.Args()
if len(args) < 1 || args[0] == "help" {
if len(args) == 1 {
usage(0)
}
if len(args) > 1 {
for _, cmd := range commands {
if cmd.Name() == args[1] {
tmpl(os.Stdout, helpTemplate, cmd)
return
}
}
}
usage(2)
}
// Commands use panic to abort execution when something goes wrong.
// Panics are logged at the point of error. Ignore those.
defer func() {
if err := recover(); err != nil {
if _, ok := err.(LoggedError); !ok {
// This panic was not expected / logged.
panic(err)
}
os.Exit(1)
}
}()
for _, cmd := range commands {
if cmd.Name() == args[0] {
cmd.Run(args[1:])
return
}
}
errorf("unknown command %q\nRun 'revel help' for usage.\n", args[0])
}
func errorf(format string, args ...interface{}) {
// Ensure the user's command prompt starts on the next line.
if !strings.HasSuffix(format, "\n") {
format += "\n"
}
fmt.Fprintf(os.Stderr, format, args...)
panic(LoggedError{}) // Panic instead of os.Exit so that deferred will run.
}
const header = `~
~ revel! http://revel.github.io
~
`
const usageTemplate = `usage: revel command [arguments]
The commands are:
{{range .}}
{{.Name | printf "%-11s"}} {{.Short}}{{end}}
Use "revel help [command]" for more information.
`
var helpTemplate = `usage: revel {{.UsageLine}}
{{.Long}}
`
func usage(exitCode int) {
tmpl(os.Stderr, usageTemplate, commands)
os.Exit(exitCode)
}
func tmpl(w io.Writer, text string, data interface{}) {
t := template.New("top")
template.Must(t.Parse(text))
if err := t.Execute(w, data); err != nil {
panic(err)
}
}
func init() {
rand.Seed(time.Now().UnixNano())
}
+142
View File
@@ -0,0 +1,142 @@
// Copyright (c) 2012-2016 The Revel Framework Authors, All rights reserved.
// Revel Framework source code and usage is governed by a MIT style
// license that can be found in the LICENSE file.
package main
import (
"go/build"
"strconv"
"github.com/revel/cmd/harness"
"github.com/revel/revel"
)
var cmdRun = &Command{
UsageLine: "run [import path] [run mode] [port]",
Short: "run a Revel application",
Long: `
Run the Revel web application named by the given import path.
For example, to run the chat room sample application:
revel run github.com/revel/examples/chat dev
The run mode is used to select which set of app.conf configuration should
apply and may be used to determine logic in the application itself.
Run mode defaults to "dev".
You can set a port as an optional third parameter. For example:
revel run github.com/revel/examples/chat prod 8080`,
}
// RunArgs holds revel run parameters
type RunArgs struct {
ImportPath string
Mode string
Port int
}
func init() {
cmdRun.Run = runApp
}
func parseRunArgs(args []string) *RunArgs {
inputArgs := RunArgs{
ImportPath: importPathFromCurrentDir(),
Mode: DefaultRunMode,
Port: revel.HTTPPort,
}
switch len(args) {
case 3:
// Possible combinations
// revel run [import-path] [run-mode] [port]
port, err := strconv.Atoi(args[2])
if err != nil {
errorf("Failed to parse port as integer: %s", args[2])
}
inputArgs.ImportPath = args[0]
inputArgs.Mode = args[1]
inputArgs.Port = port
case 2:
// Possible combinations
// 1. revel run [import-path] [run-mode]
// 2. revel run [import-path] [port]
// 3. revel run [run-mode] [port]
if _, err := build.Import(args[0], "", build.FindOnly); err == nil {
// 1st arg is the import path
inputArgs.ImportPath = args[0]
if port, err := strconv.Atoi(args[1]); err == nil {
// 2nd arg is the port number
inputArgs.Port = port
} else {
// 2nd arg is the run mode
inputArgs.Mode = args[1]
}
} else {
// 1st arg is the run mode
port, err := strconv.Atoi(args[1])
if err != nil {
errorf("Failed to parse port as integer: %s", args[1])
}
inputArgs.Mode = args[0]
inputArgs.Port = port
}
case 1:
// Possible combinations
// 1. revel run [import-path]
// 2. revel run [port]
// 3. revel run [run-mode]
_, err := build.Import(args[0], "", build.FindOnly)
if err != nil {
revel.RevelLog.Warn("Unable to run using an import path, assuming import path is working directory %s %s", "Argument", args[0], "error", err.Error())
}
println("Trying to build with", args[0], err)
if err == nil {
// 1st arg is the import path
inputArgs.ImportPath = args[0]
} else if port, err := strconv.Atoi(args[0]); err == nil {
// 1st arg is the port number
inputArgs.Port = port
} else {
// 1st arg is the run mode
inputArgs.Mode = args[0]
}
}
return &inputArgs
}
func runApp(args []string) {
runArgs := parseRunArgs(args)
// Find and parse app.conf
revel.Init(runArgs.Mode, runArgs.ImportPath, "")
revel.LoadMimeConfig()
// fallback to default port
if runArgs.Port == 0 {
runArgs.Port = revel.HTTPPort
}
revel.RevelLog.Infof("Running %s (%s) in %s mode\n", revel.AppName, revel.ImportPath, runArgs.Mode)
revel.RevelLog.Debug("Base path:", "path", revel.BasePath)
// If the app is run in "watched" mode, use the harness to run it.
if revel.Config.BoolDefault("watch", true) && revel.Config.BoolDefault("watch.code", true) {
revel.RevelLog.Debug("Running in watched mode.")
revel.HTTPPort = runArgs.Port
harness.NewHarness().Run() // Never returns.
}
// Else, just build and run the app.
revel.RevelLog.Debug("Running in live build mode.")
app, err := harness.Build()
if err != nil {
errorf("Failed to build app: %s", err)
}
app.Port = runArgs.Port
app.Cmd().Run()
}
+3
View File
@@ -0,0 +1,3 @@
test-results/
tmp/
routes/
+43
View File
@@ -0,0 +1,43 @@
# Welcome to Revel
A high-productivity web framework for the [Go language](http://www.golang.org/).
### Start the web server:
revel run myapp
### Go to http://localhost:9000/ and you'll see:
"It works"
## Code Layout
The directory structure of a generated Revel application:
conf/ Configuration directory
app.conf Main app configuration file
routes Routes definition file
app/ App sources
init.go Interceptor registration
controllers/ App controllers go here
views/ Templates directory
messages/ Message files
public/ Public static assets
css/ CSS files
js/ Javascript files
images/ Image files
tests/ Test suites
## Help
* The [Getting Started with Revel](http://revel.github.io/tutorial/gettingstarted.html).
* The [Revel guides](http://revel.github.io/manual/index.html).
* The [Revel sample apps](http://revel.github.io/examples/index.html).
* The [API documentation](https://godoc.org/github.com/revel/revel).
+13
View File
@@ -0,0 +1,13 @@
package controllers
import (
"github.com/revel/revel"
)
type App struct {
*revel.Controller
}
func (c App) Index() revel.Result {
return c.Render()
}
+57
View File
@@ -0,0 +1,57 @@
package app
import (
"github.com/revel/revel"
)
var (
// AppVersion revel app version (ldflags)
AppVersion string
// BuildTime revel app build-time (ldflags)
BuildTime string
)
func init() {
// Filters is the default set of global filters.
revel.Filters = []revel.Filter{
revel.PanicFilter, // Recover from panics and display an error page instead.
revel.RouterFilter, // Use the routing table to select the right Action
revel.FilterConfiguringFilter, // A hook for adding or removing per-Action filters.
revel.ParamsFilter, // Parse parameters into Controller.Params.
revel.SessionFilter, // Restore and write the session cookie.
revel.FlashFilter, // Restore and write the flash cookie.
revel.ValidationFilter, // Restore kept validation errors and save new ones from cookie.
revel.I18nFilter, // Resolve the requested language
HeaderFilter, // Add some security based headers
revel.InterceptorFilter, // Run interceptors around the action.
revel.CompressFilter, // Compress the result.
revel.ActionInvoker, // Invoke the action.
}
// Register startup functions with OnAppStart
// revel.DevMode and revel.RunMode only work inside of OnAppStart. See Example Startup Script
// ( order dependent )
// revel.OnAppStart(ExampleStartupScript)
// revel.OnAppStart(InitDB)
// revel.OnAppStart(FillCache)
}
// HeaderFilter adds common security headers
// There is a full implementation of a CSRF filter in
// https://github.com/revel/modules/tree/master/csrf
var HeaderFilter = func(c *revel.Controller, fc []revel.Filter) {
c.Response.Out.Header().Add("X-Frame-Options", "SAMEORIGIN")
c.Response.Out.Header().Add("X-XSS-Protection", "1; mode=block")
c.Response.Out.Header().Add("X-Content-Type-Options", "nosniff")
fc[0](c, fc[1:]) // Execute the next filter stage.
}
//func ExampleStartupScript() {
// // revel.DevMod and revel.RunMode work here
// // Use this script to check for dev mode and set dev/prod startup scripts here!
// if revel.DevMode == true {
// // Dev mode
// }
//}
+21
View File
@@ -0,0 +1,21 @@
{{set . "title" "Home"}}
{{template "header.html" .}}
<header class="jumbotron" style="background-color:#A9F16C">
<div class="container">
<div class="row">
<h1>It works!</h1>
<p></p>
</div>
</div>
</header>
<div class="container">
<div class="row">
<div class="span6">
{{template "flash.html" .}}
</div>
</div>
</div>
{{template "footer.html" .}}
+64
View File
@@ -0,0 +1,64 @@
<style type="text/css">
#sidebar {
position: absolute;
right: 0px;
top:69px;
max-width: 75%;
z-index: 1000;
background-color: #fee;
border: thin solid grey;
padding: 10px;
}
#toggleSidebar {
position: absolute;
right: 0px;
top: 50px;
background-color: #fee;
}
</style>
<div id="sidebar" style="display:none;">
<h4>Available pipelines</h4>
<dl>
{{ range $index, $value := .}}
<dt>{{$index}}</dt>
<dd>{{$value}}</dd>
{{end}}
</dl>
<h4>Flash</h4>
<dl>
{{ range $index, $value := .flash}}
<dt>{{$index}}</dt>
<dd>{{$value}}</dd>
{{end}}
</dl>
<h4>Errors</h4>
<dl>
{{ range $index, $value := .errors}}
<dt>{{$index}}</dt>
<dd>{{$value}}</dd>
{{end}}
</dl>
</div>
<a id="toggleSidebar" href="#" class="toggles"><i class="glyphicon glyphicon-chevron-left"></i></a>
<script>
$sidebar = 0;
$('#toggleSidebar').click(function() {
if ($sidebar === 1) {
$('#sidebar').hide();
$('#toggleSidebar i').addClass('glyphicon-chevron-left');
$('#toggleSidebar i').removeClass('glyphicon-chevron-right');
$sidebar = 0;
}
else {
$('#sidebar').show();
$('#toggleSidebar i').addClass('glyphicon-chevron-right');
$('#toggleSidebar i').removeClass('glyphicon-chevron-left');
$sidebar = 1;
}
return false;
});
</script>
+20
View File
@@ -0,0 +1,20 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>Not found</title>
</head>
<body>
{{if eq .RunMode "dev"}}
{{template "errors/404-dev.html" .}}
{{else}}
{{with .Error}}
<h1>
{{.Title}}
</h1>
<p>
{{.Description}}
</p>
{{end}}
{{end}}
</body>
</html>
+16
View File
@@ -0,0 +1,16 @@
<!DOCTYPE html>
<html>
<head>
<title>Application error</title>
</head>
<body>
{{if eq .RunMode "dev"}}
{{template "errors/500-dev.html" .}}
{{else}}
<h1>Oops, an error occured</h1>
<p>
This exception has been logged.
</p>
{{end}}
</body>
</html>
+18
View File
@@ -0,0 +1,18 @@
{{if .flash.success}}
<div class="alert alert-success">
{{.flash.success}}
</div>
{{end}}
{{if or .errors .flash.error}}
<div class="alert alert-danger">
{{if .flash.error}}
{{.flash.error}}
{{end}}
<ul style="margin-top:10px;">
{{range .errors}}
<li>{{.}}</li>
{{end}}
</ul>
</div>
{{end}}
+5
View File
@@ -0,0 +1,5 @@
{{if eq .RunMode "dev"}}
{{template "debug.html" .}}
{{end}}
</body>
</html>
+19
View File
@@ -0,0 +1,19 @@
<!DOCTYPE html>
<html>
<head>
<title>{{.title}}</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="/public/css/bootstrap-3.3.6.min.css">
<link rel="shortcut icon" type="image/png" href="/public/img/favicon.png">
<script src="/public/js/jquery-2.2.4.min.js"></script>
<script src="/public/js/bootstrap-3.3.6.min.js"></script>
{{range .moreStyles}}
<link rel="stylesheet" type="text/css" href="/public/{{.}}">
{{end}}
{{range .moreScripts}}
<script src="/public/{{.}}" type="text/javascript" charset="utf-8"></script>
{{end}}
</head>
<body>
+207
View File
@@ -0,0 +1,207 @@
################################################################################
# Revel configuration file
# More info at http://revel.github.io/manual/appconf.html
################################################################################
# Sets `revel.AppName` for use in-app.
# Example:
# `if revel.AppName {...}`
app.name = {{ .AppName }}
# A secret string which is passed to cryptographically sign the cookie to prevent
# (and detect) user modification.
# Keep this string secret or users will be able to inject arbitrary cookie values
# into your application
app.secret = {{ .Secret }}
# Revel running behind proxy like nginx, haproxy, etc.
app.behind.proxy = false
# The IP address on which to listen.
http.addr =
# The port on which to listen.
http.port = 9000
# Whether to use SSL or not.
http.ssl = false
# Path to an X509 certificate file, if using SSL.
#http.sslcert =
# Path to an X509 certificate key, if using SSL.
#http.sslkey =
# Timeout specifies a time limit for request (in seconds) made by a single client.
# A Timeout of zero means no timeout.
http.timeout.read = 90
http.timeout.write = 60
# For any cookies set by Revel (Session,Flash,Error) these properties will set
# the fields of:
# http://golang.org/pkg/net/http/#Cookie
#
# Each cookie set by Revel is prefixed with this string.
cookie.prefix = REVEL
# A secure cookie has the secure attribute enabled and is only used via HTTPS,
# ensuring that the cookie is always encrypted when transmitting from client to
# server. This makes the cookie less likely to be exposed to cookie theft via
# eavesdropping.
#
# Defaults to false. If 'http.ssl' is enabled, this will be defaulted to true.
# This should only be true when Revel is handling SSL connections. If you are
# using a proxy in front of revel (Nginx, Apache, etc), then this should be left
# as false.
# cookie.secure = false
# Limit cookie access to a given domain.
#cookie.domain =
# Define when your session cookie expires.
# Values:
# "720h"
# A time duration (http://golang.org/pkg/time/#ParseDuration) after which
# the cookie expires and the session is invalid.
# "session"
# Sets a session cookie which invalidates the session when the user close
# the browser.
session.expires = 720h
# The date format used by Revel. Possible formats defined by the Go `time`
# package (http://golang.org/pkg/time/#Parse)
format.date = 2006-01-02
format.datetime = 2006-01-02 15:04
# Determines whether the template rendering should use chunked encoding.
# Chunked encoding can decrease the time to first byte on the client side by
# sending data before the entire template has been fully rendered.
results.chunked = false
# The default language of this application.
i18n.default_language = en
# The default format when message is missing.
# The original message shows in %s
#i18n.unknown_format = "??? %s ???"
# Module to serve static content such as CSS, JavaScript and Media files
# Allows Routes like this:
# `Static.ServeModule("modulename","public")`
module.static = github.com/revel/modules/static
################################################################################
# Section: dev
# This section is evaluated when running Revel in dev mode. Like so:
# `revel run path/to/myapp`
[dev]
# This sets `revel.DevMode` for use in-app.
# Example:
# `if revel.DevMode {...}`
# or in your templates with
# `{{.DevMode}}`
# Values:
# "true"
# Sets `DevMode` to `true`.
# "false"
# Sets `DevMode` to `false`.
mode.dev = true
# Pretty print JSON/XML when calling RenderJSON/RenderXML
# Values:
# "true"
# Enables pretty printing.
# "false"
# Disables pretty printing.
results.pretty = true
# Watch your applicaton files for changes and automatically rebuild
# Values:
# "true"
# Enables auto rebuilding.
# "false"
# Disables auto rebuilding.
watch = true
# Define when to rebuild new changes.
# Values:
# "normal"
# Rebuild when a new request is received and changes have been detected.
# "eager"
# Rebuild as soon as changes are detected.
watch.mode = normal
# Watch the entire `$GOPATH` for changes.
# Values:
# "true"
# Includes `$GOPATH` in watch path.
# "false"
# Excludes `$GOPATH` from watch path. Default value.
#watch.gopath = true
# Module to run code tests in the browser
# See:
# http://revel.github.io/manual/testing.html
module.testrunner = github.com/revel/modules/testrunner
# Where to log the various Revel logs
# Values:
# "off"
# Disable log output.
# "stdout"
# Log to OS's standard output.
# "stderr"
# Log to Os's standard error output. Default value.
# "relative/path/to/log"
# Log to file.
log.all.filter.module.app = stdout # Log all loggers for the application to the stdout
log.error.nfilter.module.app = stderr # Everything else that logs an error to stderr
log.crit.output = stderr # Everything that logs something as critical goes to this
# Revel request access log
# Access log line format:
# INFO 21:53:55 static server-engine.go:169: Request Stats ip=127.0.0.1 path=/public/vendors/datatables.net-buttons/js/buttons.html5.min.js method=GET start=2017/08/31 21:53:55 status=200 duration_seconds=0.0002583 section=requestlog
log.request.output = stdout
################################################################################
# Section: prod
# This section is evaluated when running Revel in production mode. Like so:
# `revel run path/to/myapp prod`
# See:
# [dev] section for documentation of the various settings
[prod]
mode.dev = false
results.pretty = false
watch = false
module.testrunner =
log.warn.output = log/%(app.name)-warn.json # Log all warn messages to file
log.error.output = log/%(app.name)-error.json # Log all errors to file
log.crit.output = log/%(app.name)-critical.json # Log all critical to file
# Revel request access log (json format)
# Example:
# log.request.output = %(app.name)s-request.json
log.request.output = log/%(app.name)s-requests.json
+26
View File
@@ -0,0 +1,26 @@
# Routes Config
#
# This file defines all application routes (Higher priority routes first)
#
module:testrunner
# module:jobs
GET / App.Index
# Ignore favicon requests
GET /favicon.ico 404
# Map static resources from the /app/public folder to the /public path
GET /public/*filepath Static.Serve("public")
# Catch all, this will route any request into the controller path
#
# **** WARNING ****
# Enabling this exposes any controller and function to the web.
# ** This is a serious security issue if used online **
#
# For rapid development uncomment the following to add new controller.action endpoints
# without having to add them to the routes table.
# * /:controller/:action :controller.:action
+7
View File
@@ -0,0 +1,7 @@
# Sample messages file for the English language (en)
# Message file extensions should be ISO 639-1 codes (http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)
# Sections within each message file can optionally override the defaults using ISO 3166-1 alpha-2 codes (http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)
# See also:
# - http://www.rfc-editor.org/rfc/bcp/bcp47.txt
# - http://www.w3.org/International/questions/qa-accept-lang-locales
@@ -0,0 +1,5 @@
/*!
* Bootstrap v3.3.6 (http://getbootstrap.com)
* Copyright 2011-2015 Twitter, Inc.
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
*//*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}article,aside,details,figcaption,figure,footer,header,hgroup,main,menu,nav,section,summary{display:block}audio,canvas,progress,video{display:inline-block;vertical-align:baseline}audio:not([controls]){display:none;height:0}[hidden],template{display:none}a{background-color:transparent}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}dfn{font-style:italic}h1{margin:.67em 0;font-size:2em}mark{color:#000;background:#ff0}small{font-size:80%}sub,sup{position:relative;font-size:75%;line-height:0;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}img{border:0}svg:not(:root){overflow:hidden}figure{margin:1em 40px}hr{height:0;-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box}pre{overflow:auto}code,kbd,pre,samp{font-family:monospace,monospace;font-size:1em}button,input,optgroup,select,textarea{margin:0;font:inherit;color:inherit}button{overflow:visible}button,select{text-transform:none}button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer}button[disabled],html input[disabled]{cursor:default}button::-moz-focus-inner,input::-moz-focus-inner{padding:0;border:0}input{line-height:normal}input[type=checkbox],input[type=radio]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box;padding:0}input[type=number]::-webkit-inner-spin-button,input[type=number]::-webkit-outer-spin-button{height:auto}input[type=search]{-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box;-webkit-appearance:textfield}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}fieldset{padding:.35em .625em .75em;margin:0 2px;border:1px solid silver}legend{padding:0;border:0}textarea{overflow:auto}optgroup{font-weight:700}table{border-spacing:0;border-collapse:collapse}td,th{padding:0}/*! Source: https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css */@media print{*,:after,:before{color:#000!important;text-shadow:none!important;background:0 0!important;-webkit-box-shadow:none!important;box-shadow:none!important}a,a:visited{text-decoration:underline}a[href]:after{content:" (" attr(href) ")"}abbr[title]:after{content:" (" attr(title) ")"}a[href^="javascript:"]:after,a[href^="#"]:after{content:""}blockquote,pre{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}h2,h3,p{orphans:3;widows:3}h2,h3{page-break-after:avoid}.navbar{display:none}.btn>.caret,.dropup>.btn>.caret{border-top-color:#000!important}.label{border:1px solid #000}.table{border-collapse:collapse!important}.table td,.table th{background-color:#fff!important}.table-bordered td,.table-bordered th{border:1px solid #ddd!important}}@font-face{font-family:'Glyphicons Halflings';src:url(../fonts/glyphicons-halflings-regular.eot);src:url(../fonts/glyphicons-halflings-regular.eot?#iefix) format('embedded-opentype'),url(../fonts/glyphicons-halflings-regular.woff2) format('woff2'),url(../fonts/glyphicons-halflings-regular.woff) format('woff'),url(../fonts/glyphicons-halflings-regular.ttf) format('truetype'),url(../fonts/glyphicons-halflings-regular.svg#glyphicons_halflingsregular) format('svg')}.glyphicon{position:relative;top:1px;display:inline-block;font-family:'Glyphicons Halflings';font-style:normal;font-weight:400;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.glyphicon-asterisk:before{content:"\002a"}.glyphicon-plus:before{content:"\002b"}.glyphicon-eur:before,.glyphicon-euro:before{content:"\20ac"}.glyphicon-minus:before{content:"\2212"}.glyphicon-cloud:before{content:"\2601"}.glyphicon-envelope:before{content:"\2709"}.glyphicon-pencil:before{content:"\270f"}.glyphicon-glass:before{content:"\e001"}.glyphicon-music:before{content:"\e002"}.glyphicon-search:before{content:"\e003"}.glyphicon-heart:before{content:"\e005"}.glyphicon-star:before{content:"\e006"}.glyphicon-star-empty:before{content:"\e007"}.glyphicon-user:before{content:"\e008"}.glyphicon-film:before{content:"\e009"}.glyphicon-th-large:before{content:"\e010"}.glyphicon-th:before{content:"\e011"}.glyphicon-th-list:before{content:"\e012"}.glyphicon-ok:before{content:"\e013"}.glyphicon-remove:before{content:"\e014"}.glyphicon-zoom-in:before{content:"\e015"}.glyphicon-zoom-out:before{content:"\e016"}.glyphicon-off:before{content:"\e017"}.glyphicon-signal:before{content:"\e018"}.glyphicon-cog:before{content:"\e019"}.glyphicon-trash:before{content:"\e020"}.glyphicon-home:before{content:"\e021"}.glyphicon-file:before{content:"\e022"}.glyphicon-time:before{content:"\e023"}.glyphicon-road:before{content:"\e024"}.glyphicon-download-alt:before{content:"\e025"}.glyphicon-download:before{content:"\Line truncated
Loaded 100 of 670 files, more files were not shown because too many files have changed in this diff. Show more